DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 08/01/2025 have been considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 2-3, 8, and 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over OI et al. (US Pub. No. 2008/0304707 A1) in view of Cui (US Pub. No. 2021/0318123 A1).
Regarding claim 2, OI discloses an image analysis method comprising:
a. capturing a first plurality of images that include an object of interest with an image capturing device over a period of time; b. detecting the object of interest in each of the images with an object detection model; (See OI ¶50, “A camera video is used as at least an input to the image-recognition processing unit 102 and the self-position detecting unit 103. The image-recognition processing unit 102 outputs recognition result including position and posture information in images of various objects (detection targets) photographed by the camera 101 to the data constructing unit 110.
Further see OI ¶52, “As image recognition processing executed by the image-recognition processing unit 102, image recognition processing based on image recognition processing disclosed in, for example, JP-A-2006-190191 and JP-A-2006-190192 is executed.”)
d. approximating the location of the object of interest from the stored first plurality of images, the approximated location comprising the learned location of the object of interest; e. storing the learned location of the object of interest in a persistence object database; (See OI ¶51, “The data constructing unit 110 has an environmental map updating unit 111 that is inputted with the recognition result from the image-recognition processing unit 102 and the camera position and posture information from the self-position detecting unit 103, calculates positions and postures of various objects in the world coordinate system (the coordinate system of the environment map), and updates the environmental map and an environmental map database 112 that stores the updated environmental map. The environmental map created by the data constructing unit 110 is data including detailed information concerning various objects included in an image photographed by the camera 101, for example, various objects such as a "table" and a "chair", specifically, detailed information such as a three-dimensional shape and position and posture information.”)
f. capturing an additional plurality of images that include the object of interest with an image capturing device after the learned location of the object of interest has been stored in the persistent object database; (See OI ¶97, “As shown in FIG. 9, the information processing apparatus includes the camera 101 that photographs a peripheral environment,”)
h. approximating the location of the object of interest from the additional plurality of images; (See OI ¶97, “the image-recognition processing unit 102 that is inputted with a photographed image of the camera 101 and performs image recognition, the self-position detecting unit 103 that is inputted with the photographed image of the camera 101 and estimates a position and a posture of the camera 101,”)
i. determining whether the approximated location of the object of interest from the additional plurality of images deviates from the learned location of the object of interest in the persistence object database beyond a defined threshold; j. wherein if the approximated location of the object of interest from the additional plurality of images deviates from the learned location beyond a defined threshold, storing the approximated location of the object of interest from the additional plurality of images in the persistence object database as the learned location of the object of interest. (See OI ¶107-108, “The feature point-comparing unit 522 calculates positions of matched feature points and a distance between the feature points. Information concerning the positions and the distance is outputted to the intra-image change-area extracting unit 523 of the environmental-map updating unit 511. [0108] The intra-image change-area extracting unit 523 extracts, as an update area of the environmental map, an area other than an area where the distance between the matched feature points is smaller than a threshold set in advance. Information concerning the extracted area is outputted to the environmental map updating unit 524.”)
OI discloses the above limitations, but he fails to disclose, c. storing the plurality of images in a database: g. storing the additional plurality of images in a database.
However, Cui discloses, c. storing the plurality of images in a database; g. storing the additional plurality of images in a database; (See Cui ¶35, “The updating engine 120 may update one or more maps in the vehicle 102, such as HD maps and/or satellite maps, with the current data of the particular location or the particular object. For example, the updating engine 120 may upload an image of the current data to replace a previous corresponding image of a same object, feature, or location in the map data.” Further see Cui ¶84, “The data stores described herein may be any suitable structure (e.g., an active database, a relational database, a self-referential database.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the storing images in map when updating a map as suggested by Cui to OI’s environmental map database using known engineering techniques, with a reasonable expectation of success. The motivation for doing so is because standard maps show lines and markers, but adding images helps users identify landmarks, entrances, and destinations visually, reducing navigation errors.
Regarding claim 3, OI AND Cui disclose, the image analysis method of claim 2 further comprising repeating steps f-j a plurality of times. (See OI ¶4, “The present invention relates to an information processing apparatus, an information processing method, and a computer program for observing a moving environment in an agent (a moving object) such as a robot including a camera, executing creation of a map (an environmental map) (mapping) around the agent, i.e., environmental map creation processing according to an observation state.” Whereby the steps must be repeated, since the map is constantly being updated as the robot moves through the environment.)
Regarding claim 8, OI AND Cui disclose, the image analysis of claim 2, further comprising repeating steps a-j for a plurality of objects of interest. (See OI ¶4, “The present invention relates to an information processing apparatus, an information processing method, and a computer program for observing a moving environment in an agent (a moving object) such as a robot including a camera, executing creation of a map (an environmental map) (mapping) around the agent, i.e., environmental map creation processing according to an observation state.” Whereby the steps must be repeated, since the map is constantly being updated as the robot moves through the environment.)
Regarding claim 10, OI discloses, an image analysis system comprising: an image capturing device; (See OI ¶49, “As shown in FIG. 1, the information processing apparatus includes a camera 101 that photographs a peripheral environment.”)
one or more processors operably coupled to the image capturing device; one or more computer readable storage media operably coupled to at least one of the processors; program instructions stored on the one or more computer readable storage media for execution by at least one of the processors to cause the image analysis system to perform the actions of: (See OI ¶131, “When the processing by the software is executed, it is possible to install a program having a processing sequence recorded therein in a memory in a computer built in dedicated hardware and cause the computer to execute the program or install the program in a general-purpose computer capable of executing various kinds of processing and cause the computer to execute the program. For example, the program can be recorded in a recording medium in advance.”)
one or more persistence object databases having the learned location of a plurality of objects of interest that have obtained persistence stored thereon; (See OI ¶51, “the environmental map and an environmental map database 112 that stores the updated environmental map. The environmental map created by the data constructing unit 110 is data including detailed information concerning various objects included in an image photographed by the camera 101, for example, various objects such as a "table" and a "chair", specifically, detailed information such as a three-dimensional shape and position and posture information.”)
a. capturing, with an image capturing device, a first plurality of images that include an object of interest that is stored in at least one of the persistence object databases; (See OI ¶97, “As shown in FIG. 9, the information processing apparatus includes the camera 101 that photographs a peripheral environment,”)
b. detecting the object of interest in each of the images with an object detection model; d. approximating the location of the object of interest from the first plurality of images stored in memory; (See OI ¶97, “the image-recognition processing unit 102 that is inputted with a photographed image of the camera 101 and performs image recognition, the self-position detecting unit 103 that is inputted with the photographed image of the camera 101 and estimates a position and a posture of the camera 101,”)
e. determining whether the approximated location of the object of interest from the first plurality of images deviates from the learned location of the object of interest in the persistence object database beyond a defined threshold; and f. wherein if the approximated location of the object of interest from the first plurality of images deviates from the learned location beyond a defined threshold, storing the approximated location of the object of interest from the first plurality of images in the persistence object database as the learned location of the object of interest. (See OI ¶107-108, “The feature point-comparing unit 522 calculates positions of matched feature points and a distance between the feature points. Information concerning the positions and the distance is outputted to the intra-image change-area extracting unit 523 of the environmental-map updating unit 511. [0108] The intra-image change-area extracting unit 523 extracts, as an update area of the environmental map, an area other than an area where the distance between the matched feature points is smaller than a threshold set in advance. Information concerning the extracted area is outputted to the environmental map updating unit 524.”)
OI discloses the above limitations, but he fails to disclose, c. storing the plurality of images in memory;
However, Cui discloses, c. storing the plurality of images in memory; (See Cui ¶35, “The updating engine 120 may update one or more maps in the vehicle 102, such as HD maps and/or satellite maps, with the current data of the particular location or the particular object. For example, the updating engine 120 may upload an image of the current data to replace a previous corresponding image of a same object, feature, or location in the map data.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the storing images in map when updating a map as suggested by Cui to OI’s environmental map database using known engineering techniques, with a reasonable expectation of success. The motivation for doing so is because standard maps show lines and markers, but adding images helps users identify landmarks, entrances, and destinations visually, reducing navigation errors.
Regarding claim 11, OI AND Cui disclose, the image analysis system of claim 10, wherein the program instructions stored on the one or more computer readable storage media further cause the image analysis system to perform the actions of repeating steps a-f a plurality of times. (See the rejection of claim 8 as it is equally applicable for claim 11 as well.)
Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over OI et al. (US Pub. No. 2008/0304707 A1) in view of Cui (US Pub. No. 2021/0318123 A1) and in further view of Koivisto et al. (US Pub. No. 2019/0258878 A1).
Regarding claim 4, OI and Cui disclose the image analysis method of claim 2, but they fail to disclose, wherein the approximating steps comprise: clustering the approximate locations of the object of interest in the first and the additional plurality of images; and calculating a geometric average of the cluster.
However, Koivisto discloses, wherein the approximating steps comprise: clustering the approximate locations of the object of interest in the first and the additional plurality of images; (See Koivisto ¶63, “In some examples, for a single image 204 or frame, or a set of images or frames, each spatial element of the object detector 106 may provide the detected object data for a corresponding detected object.”
Further see Koivisto ¶73, “The detected object clusterer 108 may form any number of clusters of detected objects by applying a clustering algorithm(s) to the outputs of the object detector 106 for the detected objects (e.g., after filtering). For example, the clusters of detected objects may respectively correspond to detected object regions 260A, 260B, 260C, and 260D in FIG. 2B. To cluster detected objects, the detected object clusterer 108 may cluster the locations (e.g. detected object regions) of the detected objects together.”)
and calculating a geometric average of the cluster. (See Koivisto ¶76, “The object detection system 100 may determine a location(s) for the aggregated detected object, based at least in part on the locations of the detected objects in the cluster. For example, the location may be an average and/or weighted average (e.g., by coverage value) of the locations.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the clustering detected object locations to determine the average location as suggested by Koivisto to OI and Cui’s recognizing the location of an object. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so is to filter out spatial noise and isolate the true position of a static or object of interest. Clustering groups the valid, high-confidence detections together while leaving random, isolated errors outside the main clusters as noise.
Regarding claim 5, OI, Cui, and Koivisto disclose, the image analysis method of claim 4, the clustering step comprising grouping the detected objects of interest in the plurality of images by a centroid for the detected object of interest in the plurality of images. (See Koivisto ¶87, “The location data 310 may represent any number of location coordinates defining a detected object region, such as four coordinates that correspond to the corners of a bounding box region. The location coordinates may be provided in various possible formats, such as left, bottom, top, and right coordinates. Another example of a format includes center coordinates (e.g., x, y) along with the width and height of the spatial element region.”)
Claims 6-7 and 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over OI et al. (US Pub. No. 2008/0304707 A1) in view of Cui (US Pub. No. 2021/0318123 A1) and in further view of Garrido et al. (US Pat. No. 10,277,829 B1).
Regarding claim 6, OI and Cui disclose, the image analysis method of claim 2, but they fail to disclose, further comprising, prior to the approximating step for the first plurality of images, determining whether the first plurality of images includes a predetermined minimum number of images taken in lighting that exceeds a minimum lighting threshold.
However, Garrido discloses, further comprising, prior to the approximating step for the first plurality of images, determining whether the first plurality of images includes a predetermined minimum number of images taken in lighting that exceeds a minimum lighting threshold. (See Garrido 10:16-19, “In still other embodiments the determined illumination value may be calculated using some form of a moving average algorithm, e.g., calculating an average luminance of the N most recently obtained image frames.”
Further see Garrido 10:50-54, “Once the frame illumination value has been determined in the desired fashion, the process may proceed to Step 505, wherein the device may evaluate whether the illumination value is greater than a predetermined (or dynamic) threshold value.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include obtaining video based on thresholding the illumination value of N frames as suggested by Garrido to OI and Cui’s use to video frames to recognize object location. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so is because in image processing, computer vision, and biometric systems, low-light frames introduce noise, motion blur, and artifacts. These issues cause algorithms to fail. Checking the lighting levels across a specific count of images early in the pipeline prevents system failures and optimizes performance
Regarding claim 7, OI, Cui, and Garrido disclose, the image analysis of claim 6, wherein if the first plurality of images includes the predetermined minimum number of images taken in lighting that exceeds a minimum lighting threshold, the approximating step comprising using the images taken in lighting that exceeds a minimum lighting threshold to approximate the location of the object of interest from the first plurality of images. (See Garrido Fig. 1, where the video capture is considered done in step 130 after the lighting criteria is done. Based on the combination of references, only this high-quality video would be input to OI object recognition module that detect object location.)
Regarding claim 12, OI, Cui, and Garrido disclose, the image analysis system of claim 10, wherein the program instructions stored on the one or more computer readable storage media further cause the image analysis system to perform the actions of, prior to the approximating step for the first plurality of images, determining whether the first plurality of images includes a predetermined minimum number of images taken in lighting that exceeds a minimum lighting threshold. (See the rejection of claim 6, as it is equally applicable for claim 13 as well.)
Regarding claim 13, OI, Cui, and Garrido disclose, the image analysis system of claim 12, wherein if the first plurality of images includes the predetermined minimum number of images taken in lighting that exceeds a minimum lighting threshold, the program instructions stored on the one or more computer readable storage media further cause the image analysis system to use the images taken in lighting that exceeds a minimum lighting threshold to approximate the location of the object of interest from the first plurality of images. (See the rejection of claim 7, as it is equally applicable for claim 13 as well.)
Claims 9 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over OI et al. (US Pub. No. 2008/0304707 A1) in view of Cui (US Pub. No. 2021/0318123 A1) in view of Fan et al. (US Pub. No. 2014/0093131 A1) in view of Tang et al. (US Pub. No. 2019/0272425 A1) and in further view of Gurajapu et al. (US Pub. No. 2020/0387120 A1).
Regarding claim 9, OI and Cui disclose the image analysis of claim 8, but they fail to disclose, further comprising; k. capturing an image in real time in which a specific object of interest is expected to be visualized; wherein, l. if the specific object of interest is visualized, matching the specific object of interest with an object of interest stored in the persistence object database; and m. linking the specific object of interest with another feature in the real time image.
However, Fan discloses, further comprising; k. capturing an image in real time in which a specific object of interest is expected to be visualized; wherein, (See Fan ¶24, “As indicated at block 54, the processing unit 24 can initially determine location and orientation of the vehicle from data provided by, for example, a GPS or other location/orientation sensors 16, 18 depicted in FIG. 1. The processing unit 24 can then download from the database 22 shown in FIG. 1 a list of the stationary objects that are expected to be detectable at the current location and orientation, as illustrated at block 56.”)
matching the specific object of interest with an object of interest stored in the persistence object database; (See Fan ¶24, “Thereafter, as indicated at block 58, processing unit 24 can also compare the scene captured from the camera with the one obtained from the database 22 utilizing the location and orientation information. Following processing of the operation indicated at block 58, a test can be performed as illustrated at block 60, to determine if scenes are matched.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the matching an object in an image at an expected location to stored objects at same location as suggested by Fan to OI and Cui objects whose location are determined. Whereby the objects with a determined location from Koivisto can be stored in the object database of OI. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so is in order to more accurately detect objects based both on location and image matching.
OI, Cui, and Fan disclose the above limitations, but they fail to fail to disclose the limitations of step l.
However Tang discloses, l. if the specific object of interest is visualized, matching the specific object of interest with an object of interest stored in the persistence object database; (See Tang ¶31, “For example, the image recognition process may go into the idle state if the frame or camera is not steady enough and the camera focus is unable to “lock on” to certain objects in the camera view. The image recognition process may also go into the idle state if there is not sufficient lighting, among other poor image conditions that prevent proper image analysis.” Further see Tang ¶32, “For example, in a visual search application, the detected object may be an object that the user wants to use as a visual search query to search for visually similar products.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the recognizing or searching a database for objects that are in an input image only when there is sufficient lighting as suggested by Tang to OI, Cui, and Fan’s capturing of images that are used for matching. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so is to ensure that the image quality is sufficient to accurately to detect and recognize the object.
OI, Cui, Fan, and Tang disclose the above limitations, but they fail to disclose the limitations of step m.
However, Gurajapu discloses, m. linking the specific object of interest with another feature in the real time image. (See Gurajapu ¶33, “The processing of the images can include outputting an image with visual identification of flare portions and smoke portions and steam portions in each image. For instance, a visual color differentiation can be applied to a flare portion and a smoke portion of the images. … In one embodiment pixels identified as flame may be colored green, pixels identified with smoke may be colored red, and pixels identified as steam may be colored purple. … Panel 302 may also show data such as date, time, flare identification, and the like as information data display 314.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include identifying a flare and its associated flame in an image as suggested by Gurajapu to OI, Cui, Fan, and Tang’s recognition of objects at a specific location. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so as disclosed by Gurajapu ¶25, “Furthermore, the system allows for multiple flare monitoring and control data to be analyzed and used to continually train the deep learning core. Multiple flares at an industrial plant may be monitored and controlled though a single system using a deep learning core.”
Regarding claim 14, OI, Cui, Fan, and Tang disclose, the image analysis system of claim 10, wherein the program instructions stored on the one or more computer readable storage media further cause the image analysis system to perform the actions of: g. capturing an image in real time in which the object of interest is expected to be visualized; wherein, h. if the object of interest is visualized, matching the object of interest with the object of interest stored in the persistence object database; and i. linking the object of interest with another feature in the real time image. (See the rejection of claim 9 as it is equally applicable for claim 14 as well.)
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
Silver et al. (US Pat. No. 9,321,461 B1) Aspects of the disclosure relate to determining whether a feature of map information. For example, data identifying an object detected in a vehicle's environment and including location coordinates is received. This information is used to identify a corresponding feature from pre-stored map information based on a map location of the corresponding feature. The corresponding feature is defined as a curve and associated with a tag identifying a type of the corresponding object. A tolerance constraint is identified based on the tag. The curve is divided into two or more-line segments. Each line segment has a first position. The first position of a line segment is changed in order to determine a second position based on the location coordinates and the tolerance constraint. A value is determined based on a comparison of the first position to the second position. This value indicates a likelihood that the corresponding feature has changed.
Haas et al. (US Pub. No. 2014/0072173 A1) A global position of an observed object is determined by obtaining a first global position of an observed object with at least one positioning device. A determination is made as to whether a set of stored visual characteristic information of at least one landmark matches a visual characteristic information set obtained from at least one captured image comprising a scene associated with the observed object. In response to the set of stored visual characteristic information matching the obtained visual characteristic information set, a second global position of the observed object is determined based on a set of stored location information associated with the at least one landmark and the first global position.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID PERLMAN whose telephone number is (571) 270-1417.
The examiner can normally be reached on Monday - Friday; 10:00am -6:30pm.
Examiner interviews are available via telephone and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is
(571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at (866) 217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call (800) 786-9199 (IN USA OR CANADA) or (571) 272-1000.
/DAVID PERLMAN/Primary Examiner, Art Unit 2673
