DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-31 are pending for examination in the application filed 09/25/2024.
Priority
Acknowledgement is made of Applicant’s claim to priority of provisional application 63/585,247, filing date 09/26/2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/01/2025 has been considered by the examiner.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: 310, 320, 330. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 7 and 23 are objected to because of the following informalities: “LoFTR” should read “local feature transformer (LoFTR)”, or similar.
Claim 26 is objected to because of the following informalities: “activat” should read “activate”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5, 14, 21, 30, and claims depending therefrom are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 5, 14, 21, and 30 recite “the plurality of images”. There is insufficient antecedent basis for “the plurality of images” in these claims.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 4, 6, 14, 17-18, 20, 22, and 30 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Ofek (US20120062748A1).
Regarding claim 1, Ofek teaches a method (Figs. 2-3) for locating a region of interest in an image captured by a camera ([0024] More specifically, the map panoramas 150 can be typically obtained by dedicated panoramic cameras mounted on dedicated panorama-capturing vehicles. [0037] The combining 285 can be such that the image scope of the traffic camera image feed 140 is correctly positioned within the larger image scope of the map panorama 153, or, more precisely, such that image features of the traffic camera image feed are overlaid over equivalent image features of the map panorama 153. [0018] Within the context of vehicular traffic, the traffic camera image feed can be transformed and integrated into one or more existing panoramas as a whole, or only certain portions of the traffic camera image feed can be displayed within the panorama, such as only the moving vehicles, or, alternatively, such as only the moving vehicles and the underlying roadway. One mechanism for deriving transformation parameters can be with line matching algorithms that can match appropriate lines from the static portions of a video camera image feed to corresponding lines in one or more panorama images. Appropriate lines can be identified via the use of filtering techniques, such as by filtering based on the direction of motion, or filtering based on pre-existing superimposed map data), the method comprising:
in a preparation stage: generating a mask indicative of the location of the region of interest in a panorama of surroundings of the camera ([0027] More specifically, as shown in FIG. 2, a sequence of images that each comprise varying coverage, such as the images 141, 142, 143, 144 and 145, which were shown individually in the system 100 of FIG. 1, can be aligned so that common elements of each image overlap. The resulting composite can then be utilized, with subsequent movements of the source video camera simply being regarded as subsequent captures of different portions of the composite. [0028] From such a composite, one analysis, indicated by the action 225, can identify the areas of the composite that exhibited motion over the predetermined amount of time during which the image feed was sampled. The areas that indicated motion can be identified in the form of a motion mask 220, utilizing existing image analysis techniques well known to those skilled in the art);
and during runtime: generating a transformation between the panorama and the captured image ([0028] Similarly, from the composite, another analysis, indicated by the action 235, can identify the areas of the composite that remained static, or constant, over the predetermined amount of time during which the image feed was sampled. Again, existing image analysis techniques well known to those skilled in the art can be utilized to identify such areas. For example, one such technique can average images from the traffic camera image feed 140 over some or all of the predetermined amount of time during which the image feed is being sampled. [0036] As in the case of the matching 250 described above, image feature matching can likewise be utilized as part of the transformation and alignment 265 to select optimal transformation parameters 260. In one embodiment, a homography can be utilized to perform the transformation and alignment 265. More specifically, lines from the average image 230 can be randomly selected and a homography can be utilized to transform the average image 230 such that the randomly selected lines match equivalent lines in the selected portion 253);
and applying the transformation to the mask to get the location of the region of interest in the captured image ([0037] Once the transformation parameters 260 have been determined, they can be utilized to transform, in essentially real-time, the traffic camera image feed 140 being received from a traffic camera 110 and integrate that transformed image into existing map panoramas. More specifically, and as shown in the exemplary system 200 of FIG. 2, the traffic camera image feed 140, being received in real-time, can be filtered and transformed, as indicated by the action 275, based on the transformation parameters 260 in the motion mask 220. The motion mask 220 can be utilized to identify those portions of the traffic camera image feed 140 that are to be integrated into the existing map panoramas. [0038] The filtered and transformed traffic camera image feed 270 can then be combined, as indicated by the action 285, with the previously selected portion 253 of a map panorama 153. The combining 285 can be such that the image scope of the traffic camera image feed 140 is correctly positioned within the larger image scope of the map panorama 153, or, more precisely, such that image features of the traffic camera image feed are overlaid over equivalent image features of the map panorama 153. Such a combination can result in an amalgamated image 280 which includes a live, or essentially live, traffic camera image feed, as a moving and dynamic video, being displayed within a portion of a map panorama 280, which can then be displayed to the user, as indicated by the action 295).
Regarding claim 2, Ofek teaches the method of claim 1. Ofek further teaches generating the panorama by stitching a plurality of images from multiple viewpoints of the camera into the panorama ([0027] More specifically, as shown in FIG. 2, a sequence of images that each comprise varying coverage, such as the images 141, 142, 143, 144 and 145, which were shown individually in the system 100 of FIG. 1, can be aligned so that common elements of each image overlap. The resulting composite can then be utilized, with subsequent movements of the source video camera simply being regarded as subsequent captures of different portions of the composite).
Regarding claim 4, Ofek teaches the method of claim 1. Ofek further teaches presenting the captured image on a display, wherein the presentation comprises a marking of the location of the region of interest in the captured image ([0038] The filtered and transformed traffic camera image feed 270 can then be combined, as indicated by the action 285, with the previously selected portion 253 of a map panorama 153. The combining 285 can be such that the image scope of the traffic camera image feed 140 is correctly positioned within the larger image scope of the map panorama 153, or, more precisely, such that image features of the traffic camera image feed are overlaid over equivalent image features of the map panorama 153. Such a combination can result in an amalgamated image 280 which includes a live, or essentially live, traffic camera image feed, as a moving and dynamic video, being displayed within a portion of a map panorama 280, which can then be displayed to the user, as indicated by the action 295).
PNG
media_image1.png
446
372
media_image1.png
Greyscale
Regarding claim 6, Ofek teaches the method of claim 1. Ofek further teaches wherein the transformation between the panorama and the captured image includes a homography between the panorama and the captured image ([0036] As in the case of the matching 250 described above, image feature matching can likewise be utilized as part of the transformation and alignment 265 to select optimal transformation parameters 260. In one embodiment, a homography can be utilized to perform the transformation and alignment 265).
Regarding claim 14, Ofek teaches the method of claim 1. Ofek further teaches actively changing viewpoints of the camera and capturing an image at each of the multiple viewpoints to get the plurality of images from the multiple viewpoints ([0027] More specifically, as shown in FIG. 2, a sequence of images that each comprise varying coverage, such as the images 141, 142, 143, 144 and 145, which were shown individually in the system 100 of FIG. 1, can be aligned so that common elements of each image overlap. The resulting composite can then be utilized, with subsequent movements of the source video camera simply being regarded as subsequent captures of different portions of the composite).
Regarding claim 17, Ofek teaches a system (Fig. 6) for locating a region of interest in an image captured by a camera ([0024] More specifically, the map panoramas 150 can be typically obtained by dedicated panoramic cameras mounted on dedicated panorama-capturing vehicles. [0037] The combining 285 can be such that the image scope of the traffic camera image feed 140 is correctly positioned within the larger image scope of the map panorama 153, or, more precisely, such that image features of the traffic camera image feed are overlaid over equivalent image features of the map panorama 153. [0018] Within the context of vehicular traffic, the traffic camera image feed can be transformed and integrated into one or more existing panoramas as a whole, or only certain portions of the traffic camera image feed can be displayed within the panorama, such as only the moving vehicles, or, alternatively, such as only the moving vehicles and the underlying roadway. One mechanism for deriving transformation parameters can be with line matching algorithms that can match appropriate lines from the static portions of a video camera image feed to corresponding lines in one or more panorama images. Appropriate lines can be identified via the use of filtering techniques, such as by filtering based on the direction of motion, or filtering based on pre-existing superimposed map data),, the system comprising:
a memory (system memory 630); and a processor (central processing unit 620) configured to:
in a preparation stage: generate a mask indicative of the location of the region of interest in a panorama of surroundings of the camera ([0027] More specifically, as shown in FIG. 2, a sequence of images that each comprise varying coverage, such as the images 141, 142, 143, 144 and 145, which were shown individually in the system 100 of FIG. 1, can be aligned so that common elements of each image overlap. The resulting composite can then be utilized, with subsequent movements of the source video camera simply being regarded as subsequent captures of different portions of the composite. [0028] From such a composite, one analysis, indicated by the action 225, can identify the areas of the composite that exhibited motion over the predetermined amount of time during which the image feed was sampled. The areas that indicated motion can be identified in the form of a motion mask 220, utilizing existing image analysis techniques well known to those skilled in the art);
and during runtime: generate a transformation between the panorama and the captured image ([0028] Similarly, from the composite, another analysis, indicated by the action 235, can identify the areas of the composite that remained static, or constant, over the predetermined amount of time during which the image feed was sampled. Again, existing image analysis techniques well known to those skilled in the art can be utilized to identify such areas. For example, one such technique can average images from the traffic camera image feed 140 over some or all of the predetermined amount of time during which the image feed is being sampled. [0036] As in the case of the matching 250 described above, image feature matching can likewise be utilized as part of the transformation and alignment 265 to select optimal transformation parameters 260. In one embodiment, a homography can be utilized to perform the transformation and alignment 265. More specifically, lines from the average image 230 can be randomly selected and a homography can be utilized to transform the average image 230 such that the randomly selected lines match equivalent lines in the selected portion 253);
and apply the transformation to the mask to get the location of the region of interest in the captured image ([0037] Once the transformation parameters 260 have been determined, they can be utilized to transform, in essentially real-time, the traffic camera image feed 140 being received from a traffic camera 110 and integrate that transformed image into existing map panoramas. More specifically, and as shown in the exemplary system 200 of FIG. 2, the traffic camera image feed 140, being received in real-time, can be filtered and transformed, as indicated by the action 275, based on the transformation parameters 260 in the motion mask 220. The motion mask 220 can be utilized to identify those portions of the traffic camera image feed 140 that are to be integrated into the existing map panoramas. [0038] The filtered and transformed traffic camera image feed 270 can then be combined, as indicated by the action 285, with the previously selected portion 253 of a map panorama 153. The combining 285 can be such that the image scope of the traffic camera image feed 140 is correctly positioned within the larger image scope of the map panorama 153, or, more precisely, such that image features of the traffic camera image feed are overlaid over equivalent image features of the map panorama 153. Such a combination can result in an amalgamated image 280 which includes a live, or essentially live, traffic camera image feed, as a moving and dynamic video, being displayed within a portion of a map panorama 280, which can then be displayed to the user, as indicated by the action 295).
Regarding claim 18, Ofek teaches the system of claim 17. Ofek further teaches wherein the processor is configured to generate the panorama by stitching a plurality of images from multiple viewpoints of the camera into the panorama ([0027] More specifically, as shown in FIG. 2, a sequence of images that each comprise varying coverage, such as the images 141, 142, 143, 144 and 145, which were shown individually in the system 100 of FIG. 1, can be aligned so that common elements of each image overlap. The resulting composite can then be utilized, with subsequent movements of the source video camera simply being regarded as subsequent captures of different portions of the composite).
Regarding claim 20, Ofek teaches the system of claim 17. Ofek further teaches comprising a display (display device 691), wherein the processor is configured to present the captured image on a display, wherein the presentation comprises a marking of the location of the region of interest in the captured image ([0038] The filtered and transformed traffic camera image feed 270 can then be combined, as indicated by the action 285, with the previously selected portion 253 of a map panorama 153. The combining 285 can be such that the image scope of the traffic camera image feed 140 is correctly positioned within the larger image scope of the map panorama 153, or, more precisely, such that image features of the traffic camera image feed are overlaid over equivalent image features of the map panorama 153. Such a combination can result in an amalgamated image 280 which includes a live, or essentially live, traffic camera image feed, as a moving and dynamic video, being displayed within a portion of a map panorama 280, which can then be displayed to the user, as indicated by the action 295).
PNG
media_image2.png
446
372
media_image2.png
Greyscale
Regarding claim 22, Ofek teaches the system of claim 17. Ofek further teaches wherein the transformation between the panorama and the captured image includes a homography between the panorama and the captured image ([0036] As in the case of the matching 250 described above, image feature matching can likewise be utilized as part of the transformation and alignment 265 to select optimal transformation parameters 260. In one embodiment, a homography can be utilized to perform the transformation and alignment 265).
Regarding claim 30, Ofek teaches the system of claim 17. Ofek further teaches wherein the processor is configured to actively change viewpoints of the camera and capture an image at each of the multiple viewpoints to get the plurality of images from the multiple viewpoints ([0027] More specifically, as shown in FIG. 2, a sequence of images that each comprise varying coverage, such as the images 141, 142, 143, 144 and 145, which were shown individually in the system 100 of FIG. 1, can be aligned so that common elements of each image overlap. The resulting composite can then be utilized, with subsequent movements of the source video camera simply being regarded as subsequent captures of different portions of the composite).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 3 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Ofek in view of Murphy (US20110279453A1).
Regarding claim 3, Ofek teaches the method of claim 1. Ofek does not explicitly teach generating a plurality of panoramas, each for a different visibility condition.
Murphy, in the same field of endeavor of image visibility analysis, teaches generating a plurality of panoramas, each for a different visibility condition ([0050] In another embodiment, the selection of the image data for the second rendering can also be based on the context information. For example, if one or more images (e.g., panoramas) are available for a given location (e.g., a day view and a night view), the application 109 can use the context information can select the most representative images based on the context information. [0017] "context information" for providing contextual details pertaining to the current environment of the user or mobile device can be sensed as well. This may include details such as current weather conditions, time of day, traffic conditions, etc., all of which can be rendered to a GUI with respect to a location-based services).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Ofek with the teachings of Murphy to generate panoramas for different visibility conditions so that "the user is presented with a user interface (e.g., an augmented reality user interface or map) that more accurately reflects actual conditions at the scene, so that the user can more easily associate features depicted in the user interface with their real-world counterparts" [0046].
Regarding claim 19, Ofek teaches the system of claim 17. Ofek does not explicitly teach wherein the processor is configured to generate a plurality of panoramas, each for a different visibility condition.
Murphy, in the same field of endeavor of image visibility analysis, teaches wherein the processor is configured to generate a plurality of panoramas, each for a different visibility condition ([0050] In another embodiment, the selection of the image data for the second rendering can also be based on the context information. For example, if one or more images (e.g., panoramas) are available for a given location (e.g., a day view and a night view), the application 109 can use the context information can select the most representative images based on the context information. [0017] "context information" for providing contextual details pertaining to the current environment of the user or mobile device can be sensed as well. This may include details such as current weather conditions, time of day, traffic conditions, etc., all of which can be rendered to a GUI with respect to a location-based services).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ofek with the teachings of Murphy to generate panoramas for different visibility conditions so that "the user is presented with a user interface (e.g., an augmented reality user interface or map) that more accurately reflects actual conditions at the scene, so that the user can more easily associate features depicted in the user interface with their real-world counterparts" [0046].
Claims 5 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Ofek in view of Picalausa (US20170201681A1).
Regarding claim 5, Ofek teaches the method of claim 1. Ofek does not explicitly teach removing areas from the plurality of images that include presentation of metadata in the plurality of images and in-filling the removed areas in the panorama.
Picalausa, in the same field of endeavor of vehicular imaging systems teaches removing areas from the plurality of images that include presentation of metadata in the plurality of images and in-filling the removed areas in the panorama ([0110] In an optional step 1750, the ISP chip stitches together the synchronized image streams to form a single composite image stream with image data generated by each of the plurality of image sensor chips. In one implementation, the single image composite stream represents the combination of all fields of view of the plurality of image sensor chips. Step 1750 may be performed by processing circuitry 1626. Without departing from the scope hereof, step 1750 may include removing some spatial portions of at least one of the image streams. For example, step 1750 may remove a time mark column from each marked image used to generate the single composite image stream. Step 1750 may also, for one or more of the marked images, utilize only a spatial subset of the marked images. In one example of step 1750, processing circuitry 1626 stitches together a plurality of synchronized streams of marked images 142 respectively generated by the plurality of image sensor chips 110. Image cleanup module 226 may remove time mark columns from the synchronized streams of marked images 142 prior to stitching).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Ofek with the teachings of Picalausa to remove areas of metadata and in-filling the areas in the panorama for "stitching together image data from marked image 142 received from different image sensor chips 110 to produce composite image data, for example representing a larger field of view than what may be achieved using a single image sensor chip 110" [0103].
Regarding claim 21, Ofek teaches the system of claim 17. Ofek does not explicitly teach wherein the processor is configured to remove areas from the plurality of images that include presentation of metadata in the plurality of images and in-filling the removed areas in the panorama.
Picalausa, in the same field of endeavor of vehicular imaging systems teaches wherein the processor is configured to remove areas from the plurality of images that include presentation of metadata in the plurality of images and in-filling the removed areas in the panorama ([0110] In an optional step 1750, the ISP chip stitches together the synchronized image streams to form a single composite image stream with image data generated by each of the plurality of image sensor chips. In one implementation, the single image composite stream represents the combination of all fields of view of the plurality of image sensor chips. Step 1750 may be performed by processing circuitry 1626. Without departing from the scope hereof, step 1750 may include removing some spatial portions of at least one of the image streams. For example, step 1750 may remove a time mark column from each marked image used to generate the single composite image stream. Step 1750 may also, for one or more of the marked images, utilize only a spatial subset of the marked images. In one example of step 1750, processing circuitry 1626 stitches together a plurality of synchronized streams of marked images 142 respectively generated by the plurality of image sensor chips 110. Image cleanup module 226 may remove time mark columns from the synchronized streams of marked images 142 prior to stitching).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ofek with the teachings of Picalausa to remove areas of metadata and in-filling the areas in the panorama for "stitching together image data from marked image 142 received from different image sensor chips 110 to produce composite image data, for example representing a larger field of view than what may be achieved using a single image sensor chip 110" [0103].
Claims 7-9, 16, and 23-25 are rejected under 35 U.S.C. 103 as being unpatentable over Ofek in view of Chen (US20110304730A1).
Regarding claim 7, Ofek teaches the method of claim 1. Ofek does not explicitly teach wherein the transformation is generated using a method selected from the list consisting of: random sample consensus (Ransac), scale-invariant feature transform (SIFT), nearest neighbor matching and LoFTR.
Chen, in the same field of endeavor of image transformation analysis, teaches wherein the transformation is generated using a method selected from the list consisting of: random sample consensus (Ransac), scale-invariant feature transform (SIFT), nearest neighbor matching and LoFTR ([0018] If the PTZ camera 1 has moved between the time of capture of the current image and the previous image, the calculation module 104 calculates the state vector of the predetermined object or point in the current image using to the state vector of the corresponding object or point in the previous image and a feature extraction algorithm. [0019] the feature extraction algorithm is a scale-invariant feature transform algorithm (SIFT), or a speeded up robust features algorithm (SURF), for example).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Ofek with the teachings of Chen for a SIFT transform to be generated so that "The control module 106 is used for aiming the PTZ camera 1, so as to align the center point of the current image with a selected target point in the monitored area, according to the state vector of the center point of the current image. The control module 106 obtains a new image of the monitored area captured by the camera lens 40" [0020].
Regarding claim 8, Ofek teaches the method of claim 1. Ofek does not explicitly teach generating a new transformation after the camera moves.
Chen, in the same field of endeavor of image transformation analysis, teaches generating a new transformation after the camera moves ([0018] In one embodiment, if the PTZ camera 1 has not moved between the time of capture of the current image and a previous image, the calculation module 104 determines that the state vector of the predetermined object or point in the current image equals the state vector of a corresponding object or point in the previous image. For example, the state vector of the corresponding object or point in the previous image is the state vector lastly recorded in the state vector table. If the PTZ camera 1 has moved between the time of capture of the current image and the previous image, the calculation module 104 calculates the state vector of the predetermined object or point in the current image using to the state vector of the corresponding object or point in the previous image and a feature extraction algorithm. For example, as illustrated in FIG. 3, three points a, b, and c in the image captured at the time k are predetermined points, the calculation module 104 tracks the three points in the image captured at the time (k-1), such as the points a', b', and c'. As the state vectors of the three points a', b', and c' are recorded in the state vector table, the calculation module 104 can calculate the state vectors of the three points a, b, and c in the image captured at the time k, according to the state vectors of the three points a, b', and c', and the feature extraction algorithm. A formula used as the feature extraction algorithm can be as follows:
PNG
media_image3.png
122
482
media_image3.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Ofek with the teachings of Chen to generate a new transform after the camera moves because "The control module 106 is used for aiming the PTZ camera 1, so as to align the center point of the current image with a selected target point in the monitored area, according to the state vector of the center point of the current image. The control module 106 obtains a new image of the monitored area captured by the camera lens 40" [0020].
Regarding claim 9, Ofek and Chen teach the method of claim 7. Ofek does not explicitly teach detecting movement of the camera by detecting a change in pan, tilt or zoom values of the camera; or using a computer vision flow-based model to detect significant movement vectors in a video stream captured by the camera.
Chen, in the same field of endeavor of image transformation analysis, teaches detecting movement of the camera by detecting a change in pan, tilt or zoom values of the camera; or using a computer vision flow-based model to detect significant movement vectors in a video stream captured by the camera
PNG
media_image4.png
509
695
media_image4.png
Greyscale
([0018] In one embodiment, if the PTZ camera 1 has not moved between the time of capture of the current image and a previous image, the calculation module 104 determines that the state vector of the predetermined object or point in the current image equals the state vector of a corresponding object or point in the previous image…If the PTZ camera 1 has moved between the time of capture of the current image and the previous image, the calculation module 104 calculates the state vector of the predetermined object or point in the current image using to the state vector of the corresponding object or point in the previous image and a feature extraction algorithm)
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Ofek with the teachings of Chen to detect movement of the camera by detecting a change in pan, tilt or zoom values of the camera because "three points a, b, and c in the image captured at the time k are predetermined points, the calculation module 104 tracks the three points in the image captured at the time (k-1), such as the points a', b', and c'. As the state vectors of the three points a', b', and c' are recorded in the state vector table, the calculation module 104 can calculate the state vectors of the three points a, b, and c in the image captured at the time k, according to the state vectors of the three points a, b', and c', and the feature extraction algorithm" [0018] and "The control module 106 is used for aiming the PTZ camera 1, so as to align the center point of the current image with a selected target point in the monitored area, according to the state vector of the center point of the current image. The control module 106 obtains a new image of the monitored area captured by the camera lens 40" [0020].
Regarding claim 16, Ofek teaches a method (Figs. 2-3) for locating a region of interest ([0018] Within the context of vehicular traffic, the traffic camera image feed can be transformed and integrated into one or more existing panoramas as a whole, or only certain portions of the traffic camera image feed can be displayed within the panorama, such as only the moving vehicles, or, alternatively, such as only the moving vehicles and the underlying roadway. One mechanism for deriving transformation parameters can be with line matching algorithms that can match appropriate lines from the static portions of a video camera image feed to corresponding lines in one or more panorama images. Appropriate lines can be identified via the use of filtering techniques, such as by filtering based on the direction of motion, or filtering based on pre-existing superimposed map data), the method comprising:
stitching a plurality of images from multiple viewpoints of a camera into a panorama ([0027] More specifically, as shown in FIG. 2, a sequence of images that each comprise varying coverage, such as the images 141, 142, 143, 144 and 145, which were shown individually in the system 100 of FIG. 1, can be aligned so that common elements of each image overlap. The resulting composite can then be utilized, with subsequent movements of the source video camera simply being regarded as subsequent captures of different portions of the composite);
generating a mask indicative of the location of the region of interest in a panorama of surroundings of the camera ([0027] More specifically, as shown in FIG. 2, a sequence of images that each comprise varying coverage, such as the images 141, 142, 143, 144 and 145, which were shown individually in the system 100 of FIG. 1, can be aligned so that common elements of each image overlap. The resulting composite can then be utilized, with subsequent movements of the source video camera simply being regarded as subsequent captures of different portions of the composite. [0028] From such a composite, one analysis, indicated by the action 225, can identify the areas of the composite that exhibited motion over the predetermined amount of time during which the image feed was sampled. The areas that indicated motion can be identified in the form of a motion mask 220, utilizing existing image analysis techniques well known to those skilled in the art);
generating a transformation between the panorama and an image captured in the first viewpoint ([0028] Similarly, from the composite, another analysis, indicated by the action 235, can identify the areas of the composite that remained static, or constant, over the predetermined amount of time during which the image feed was sampled. Again, existing image analysis techniques well known to those skilled in the art can be utilized to identify such areas. For example, one such technique can average images from the traffic camera image feed 140 over some or all of the predetermined amount of time during which the image feed is being sampled. [0036] As in the case of the matching 250 described above, image feature matching can likewise be utilized as part of the transformation and alignment 265 to select optimal transformation parameters 260. In one embodiment, a homography can be utilized to perform the transformation and alignment 265. More specifically, lines from the average image 230 can be randomly selected and a homography can be utilized to transform the average image 230 such that the randomly selected lines match equivalent lines in the selected portion 253);
and applying the transformation to the mask to get the location of the region of interest in images captured with the first viewpoint ([0037] Once the transformation parameters 260 have been determined, they can be utilized to transform, in essentially real-time, the traffic camera image feed 140 being received from a traffic camera 110 and integrate that transformed image into existing map panoramas. More specifically, and as shown in the exemplary system 200 of FIG. 2, the traffic camera image feed 140, being received in real-time, can be filtered and transformed, as indicated by the action 275, based on the transformation parameters 260 in the motion mask 220. The motion mask 220 can be utilized to identify those portions of the traffic camera image feed 140 that are to be integrated into the existing map panoramas. [0038] The filtered and transformed traffic camera image feed 270 can then be combined, as indicated by the action 285, with the previously selected portion 253 of a map panorama 153. The combining 285 can be such that the image scope of the traffic camera image feed 140 is correctly positioned within the larger image scope of the map panorama 153, or, more precisely, such that image features of the traffic camera image feed are overlaid over equivalent image features of the map panorama 153. Such a combination can result in an amalgamated image 280 which includes a live, or essentially live, traffic camera image feed, as a moving and dynamic video, being displayed within a portion of a map panorama 280, which can then be displayed to the user, as indicated by the action 295).
Ofek does not explicitly teach detecting that the camera has moved to a first viewpoint.
Chen, in the same field of endeavor of image transformation analysis, teaches detecting that the camera has moved to a first viewpoint ([0018] In one embodiment, if the PTZ camera 1 has not moved between the time of capture of the current image and a previous image, the calculation module 104 determines that the state vector of the predetermined object or point in the current image equals the state vector of a corresponding object or point in the previous image. For example, the state vector of the corresponding object or point in the previous image is the state vector lastly recorded in the state vector table. If the PTZ camera 1 has moved between the time of capture of the current image and the previous image, the calculation module 104 calculates the state vector of the predetermined object or point in the current image using to the state vector of the corresponding object or point in the previous image and a feature extraction algorithm).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Ofek with the teachings of Chen to detect that the camera has moved to a first viewpoint so that "the calculation module 104 tracks the three points in the image captured at the time (k-1), such as the points a', b', and c'. As the state vectors of the three points a', b', and c' are recorded in the state vector table, the calculation module 104 can calculate the state vectors of the three points a, b, and c in the image captured at the time k, according to the state vectors of the three points a, b', and c', and the feature extraction algorithm" [0018].
Regarding claim 23, Ofek teaches the system of claim 17. Ofek does not explicitly teach wherein the processor is configured to generate the transformation using a method selected from the list consisting of: random sample consensus (Ransac), scale-invariant feature transform (SIFT), nearest neighbor matching and LoFTR.
Chen, in the same field of endeavor of image transformation analysis, teaches wherein the processor is configured to generate the transformation using a method selected from the list consisting of: random sample consensus (Ransac), scale-invariant feature transform (SIFT), nearest neighbor matching and LoFTR ([0018] If the PTZ camera 1 has moved between the time of capture of the current image and the previous image, the calculation module 104 calculates the state vector of the predetermined object or point in the current image using to the state vector of the corresponding object or point in the previous image and a feature extraction algorithm. [0019] the feature extraction algorithm is a scale-invariant feature transform algorithm (SIFT), or a speeded up robust features algorithm (SURF), for example).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ofek with the teachings of Chen for a SIFT transform to be generated so that "The control module 106 is used for aiming the PTZ camera 1, so as to align the center point of the current image with a selected target point in the monitored area, according to the state vector of the center point of the current image. The control module 106 obtains a new image of the monitored area captured by the camera lens 40" [0020].
Regarding claim 24, Ofek teaches the system of claim 17. Ofek does not explicitly teach wherein the processor is configured to generate a new transformation after the camera moves.
Chen, in the same field of endeavor of image transformation analysis, teaches wherein the processor is configured to generate a new transformation after the camera moves ([0018] In one embodiment, if the PTZ camera 1 has not moved between the time of capture of the current image and a previous image, the calculation module 104 determines that the state vector of the predetermined object or point in the current image equals the state vector of a corresponding object or point in the previous image. For example, the state vector of the corresponding object or point in the previous image is the state vector lastly recorded in the state vector table. If the PTZ camera 1 has moved between the time of capture of the current image and the previous image, the calculation module 104 calculates the state vector of the predetermined object or point in the current image using to the state vector of the corresponding object or point in the previous image and a feature extraction algorithm. For example, as illustrated in FIG. 3, three points a, b, and c in the image captured at the time k are predetermined points, the calculation module 104 tracks the three points in the image captured at the time (k-1), such as the points a', b', and c'. As the state vectors of the three points a', b', and c' are recorded in the state vector table, the calculation module 104 can calculate the state vectors of the three points a, b, and c in the image captured at the time k, according to the state vectors of the three points a, b', and c', and the feature extraction algorithm. A formula used as the feature extraction algorithm can be as follows:
PNG
media_image3.png
122
482
media_image3.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ofek with the teachings of Chen to generate a new transform after the camera moves because "The control module 106 is used for aiming the PTZ camera 1, so as to align the center point of the current image with a selected target point in the monitored area, according to the state vector of the center point of the current image. The control module 106 obtains a new image of the monitored area captured by the camera lens 40" [0020].
Regarding claim 25, Ofek and Chen teach the system of claim 24. Ofek does not explicitly teach wherein the processor is configured to detect movement of the camera by detecting a change in pan, tilt or zoom values of the camera; or using a computer vision flow-based model to detect significant movement vectors in a video stream captured by the camera.
Chen, in the same field of endeavor of image transformation analysis, teaches wherein the processor is configured to detect movement of the camera by detecting a change in pan, tilt or zoom values of the camera; or using a computer vision flow-based model to detect significant movement vectors in a video stream captured by the camera
PNG
media_image4.png
509
695
media_image4.png
Greyscale
([0018] In one embodiment, if the PTZ camera 1 has not moved between the time of capture of the current image and a previous image, the calculation module 104 determines that the state vector of the predetermined object or point in the current image equals the state vector of a corresponding object or point in the previous image…If the PTZ camera 1 has moved between the time of capture of the current image and the previous image, the calculation module 104 calculates the state vector of the predetermined object or point in the current image using to the state vector of the corresponding object or point in the previous image and a feature extraction algorithm).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ofek with the teachings of Chen to detect movement of the camera by detecting a change in pan, tilt or zoom values of the camera because "three points a, b, and c in the image captured at the time k are predetermined points, the calculation module 104 tracks the three points in the image captured at the time (k-1), such as the points a', b', and c'. As the state vectors of the three points a', b', and c' are recorded in the state vector table, the calculation module 104 can calculate the state vectors of the three points a, b, and c in the image captured at the time k, according to the state vectors of the three points a, b', and c', and the feature extraction algorithm" [0018] and "The control module 106 is used for aiming the PTZ camera 1, so as to align the center point of the current image with a selected target point in the monitored area, according to the state vector of the center point of the current image. The control module 106 obtains a new image of the monitored area captured by the camera lens 40" [0020].
Claims 11-13 are rejected under 35 U.S.C. 103 as being unpatentable over Ofek in view of Chen and Matsunaga (WO2014016949A1).
Regarding claim 11, Ofek and Chen teach the method of claim 9. Ofek does not explicitly teach labeling the region of interest and associating objects detected within the region of interest with the label of the region of interest.
Matsunaga, in the same field of endeavor of vehicle ROI measurement, teaches labeling the region of interest and associating objects detected within the region of interest with the label of the region of interest ([Abstract] (A) is map information representing reachable ranges for a moving body such as a vehicle. The map information is divided into a plurality of regions. Regions that are filled in represent reachable ranges for the moving body. In (A), there are three linked region groups representing reachable ranges for the moving body. (B) shows a state in which a label number has been assigned to each linked region group. [pg. 11 para. 2] The detecting unit 406 detects a specific area group related to the position of the moving body from among a plurality of area groups indicating the reachable range of each moving body on the map information. Specifically, the detection unit 406 detects a specific connected region group related to the position of the moving body from among a plurality of connected region groups on the map information. The specific connected area group is, for example, a connected area group including the own vehicle position or a connected area group closest to the own vehicle position. [pg. 3 para. 4] In the image processing example 1 of FIG. 1, the position of the moving body (own vehicle position) exists in the connected region group R3).
PNG
media_image5.png
560
1227
media_image5.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Ofek with the teachings of Matsunaga to label the region of interest and associate objects detected within the region of interest with the label of the region of interest because "the image processing apparatus selects and deletes areas unnecessary for display from a reachable range that is divided into a plurality of parts, and has a predetermined number, area, perimeter, and shape… In particular, the image processing apparatus sorts out a specific area in consideration of the vehicle position" [pg. 3 para. 2].
Regarding claim 12, Ofek and Chen teach the method of claim 9. Ofek further teaches the panorama.
PNG
media_image6.png
441
506
media_image6.png
Greyscale
Ofek does not explicitly teach associating pixels in the image with latitude and longitude coordinates; and associating a detected object with latitude and longitude coordinates of the pixels that include the detected object.
Matsunaga, in the same field of endeavor of vehicle ROI measurement, teaches associating pixels in the image with latitude and longitude coordinates ([pg. 3 para. 3] FIG. 1 is an explanatory diagram of an image processing example 1 according to the first embodiment. (A) is the map information which shows the reachable range of moving bodies, such as a vehicle. The map information is divided into a plurality of areas. Each region corresponds to one or more pixels, for example. [pg. 17 para. 6-7] The dividing unit 404 according to the present embodiment divides the map data stored in the storage device…As shown in FIG. 11, the dividing unit 404 first generates longitude / latitude information (x, y) having a point group 1100 in absolute coordinates based on the longitude x and latitude y of each of a plurality of reachable points. . The origin (0, 0) of the longitude / latitude information (x, y) is at the lower left of FIG. Then, the dividing unit 404 calculates distances w1 and w2 from the longitude ofx of the current point 500 of the vehicle to the maximum longitude x_max and the minimum longitude x_min of the reachable point farthest in the longitude x direction. Further, the dividing unit 404 calculates the distances w3 and w4 from the latitude of the current point 500 of the vehicle to the maximum latitude y_max and the minimum latitude y_min of the reachable point farthest in the latitude y direction);
and associating a detected object with latitude and longitude coordinates of the pixels that include the detected object ([pg. 3 para. 3-4] FIG. 1 is an explanatory diagram of an image processing example 1 according to the first embodiment. (A) is the map information which shows the reachable range of moving bodies, such as a vehicle. The map information is divided into a plurality of areas. Each region corresponds to one or more pixels, for example. In (A), the filled area indicates the reachable range of the moving object. A region formed by connecting filled regions is referred to as a connected region group. In (A), there are three connected region groups indicating the reachable range of the moving object. (B) is the next state of (A) and shows a state where a label number is assigned to each connected region group… In the image processing example 1 of FIG. 1, the position of the moving body (own vehicle position) exists in the connected region group R3. [pg. 11 para. 2] The detecting unit 406 detects a specific area group related to the position of the moving body from among a plurality of area groups indicating the reachable range of each moving body on the map information. Specifically, the detection unit 406 detects a specific connected region group related to the position of the moving body from among a plurality of connected region groups on the map information. The specific connected area group is, for example, a connected area group including the own vehicle position or a connected area group closest to the own vehicle position. [pg. 6 para. 1-4] The camera 314 captures images inside or outside the vehicle. The image may be either a still image or a moving image. For example, the outside of the vehicle is photographed by the camera 314, and the photographed image is analyzed by the CPU 301…The GPS unit 316 receives radio waves from GPS satellites and outputs information indicating the current position of the vehicle. The output information of the GPS unit 316 is used when the CPU 301 calculates the current position of the vehicle together with output values of various sensors 317 described later. The information indicating the current position is information for specifying one point on the map data, such as latitude / longitude and altitude).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Ofek with the teachings of Matsunaga associate detected objects in the image with latitude and longitude coordinates associated with pixels because “the display control unit 410 may extract the reachable range of the mobile body based on the longitude / latitude information of the area to which the reachable identification information is given, and display the reachable range on the display unit 411" [pg. 12 para. 7] and "the image processing apparatus selects and deletes areas unnecessary for display from a reachable range that is divided into a plurality of parts, and has a predetermined number, area, perimeter, and shape… In particular, the image processing apparatus sorts out a specific area in consideration of the vehicle position" [pg. 3 para. 2].
Regarding claim 13, Ofek, Chen, and Matsunaga teach the method of claim 11. Ofek does not explicitly teach comparing the latitude and longitude coordinates of the detected object in at least two time-spaced images to measure the speed of the detected object.
Matsunaga, in the same field of endeavor of vehicle ROI measurement, teaches comparing the latitude and longitude coordinates of the detected object in at least two time-spaced images to measure the speed of the detected object ([pg. 6 para. 1-4] The camera 314 captures images inside or outside the vehicle. The image may be either a still image or a moving image. For example, the outside of the vehicle is photographed by the camera 314, and the photographed image is analyzed by the CPU 301…The GPS unit 316 receives radio waves from GPS satellites and outputs information indicating the current position of the vehicle. The output information of the GPS unit 316 is used when the CPU 301 calculates the current position of the vehicle together with output values of various sensors 317 described later. The information indicating the current position is information for specifying one point on the map data, such as latitude / longitude and altitude. Various sensors 317 output information for determining the position and behavior of the vehicle, such as a vehicle speed sensor, an acceleration sensor, an angular velocity sensor, and a tilt sensor. The output values of the various sensors 317 are used by the CPU 301 to calculate the current position of the vehicle and the amount of change in speed and direction).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Ofek with the teachings of Matsunaga to compare latitude and longitude coordinates to measure the speed so that "the calculation unit 402 estimates an estimated energy consumption amount in a predetermined section based on information about the speed of the moving body and the moving body information" [pg. 7 para. 3]
Claims 10, 15, 26, and 31 are rejected under 35 U.S.C. 103 as being unpatentable over Ofek in view of Kadar (US20230148351A1).
Regarding claim 10, Ofek teaches the method of claim 1. Ofek does not explicitly teach activating object detection schemes selectively for detecting objects included in the region of interest and not detecting objects outside of the region of interest.
Kadar, in the same field of endeavor of traffic image analysis, teaches activating object detection schemes selectively for detecting objects included in the region of interest and not detecting objects outside of the region of interest ([0011] The stopped traffic detector may be further configured to define a region of interest in the image of the scene and configure a plurality of micro-sensors in the region of interest, wherein each micro-sensor is implemented as a state machine. The stop event may be triggered if a presence of a stationary object is detected in the region of interest based at least in part on variance values of the micro-sensors within a bounding box of the stationary object. [0055] The configuration may include configuring micro-sensors (such as micro-sensors 220 as described with regards to FIG. 2), defined detection zones 736 within the scene 734 (such as region of interest 210 as described with regards to FIG. 2)).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Ofek with the teachings of Kadar to activate object detection schemes selectively detecting objects in the region of interest because "a traffic accident, stopped vehicle or fallen object may cause significant traffic congestion that may be mitigated by altering traffic control signals and/or dispatching emergency personnel…However, such systems often result in false detections/alarms due to various possible artifacts such as shadows, blooming effects due to headlights, tire-tracks in snowy or muddy landscapes, changing daylight conditions, flickering streetlights, waving trees due to winds, and other conditions that may trigger false alarms" [0005].
Regarding claim 15, Ofek teaches the method of claim 1. Ofek does not explicitly teach wherein the camera is a roadside camera.
Kadar, in the same field of endeavor of traffic image analysis, teaches wherein the camera is a roadside camera ([0052] The systems disclosed herein are capable of running on most traffic processing systems, such as edge devices associated with roadside cameras).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Ofek with the teachings of Kadar to use roadside cameras because "Traffic control systems often use sensors to detect vehicles and other objects to help mitigate traffic congestion and improve safety…Within a traffic control system, a traffic signal controller may be used to manipulate the various phases of traffic signal at an intersection and/or along a roadway to affect traffic signalization. These traffic control systems are typically positioned adjacent to the intersection/roadway they control" [0002-0003].
Regarding claim 26, Ofek teaches the system of claim 17. Ofek does not explicitly teach wherein the processor is configured to activate object detection schemes selectively for detecting objects included in the region of interest and not detecting objects outside of the region of interest.
Kadar, in the same field of endeavor of traffic image analysis, teaches wherein the processor is configured to activate object detection schemes selectively for detecting objects included in the region of interest and not detecting objects outside of the region of interest ([0011] The stopped traffic detector may be further configured to define a region of interest in the image of the scene and configure a plurality of micro-sensors in the region of interest, wherein each micro-sensor is implemented as a state machine. The stop event may be triggered if a presence of a stationary object is detected in the region of interest based at least in part on variance values of the micro-sensors within a bounding box of the stationary object. [0055] The configuration may include configuring micro-sensors (such as micro-sensors 220 as described with regards to FIG. 2), defined detection zones 736 within the scene 734 (such as region of interest 210 as described with regards to FIG. 2)).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system, of Ofek with the teachings of Kadar to activate object detection schemes selectively detecting objects in the region of interest because "a traffic accident, stopped vehicle or fallen object may cause significant traffic congestion that may be mitigated by altering traffic control signals and/or dispatching emergency personnel…However, such systems often result in false detections/alarms due to various possible artifacts such as shadows, blooming effects due to headlights, tire-tracks in snowy or muddy landscapes, changing daylight conditions, flickering streetlights, waving trees due to winds, and other conditions that may trigger false alarms" [0005].
Regarding claim 31, Ofek teaches the system of claim 17. Ofek does not explicitly teach wherein the camera is a roadside camera.
Kadar, in the same field of endeavor of traffic image analysis, teaches wherein the camera is a roadside camera ([0052] The systems disclosed herein are capable of running on most traffic processing systems, such as edge devices associated with roadside cameras).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ofek with the teachings of Kadar to use roadside cameras because "Traffic control systems often use sensors to detect vehicles and other objects to help mitigate traffic congestion and improve safety…Within a traffic control system, a traffic signal controller may be used to manipulate the various phases of traffic signal at an intersection and/or along a roadway to affect traffic signalization. These traffic control systems are typically positioned adjacent to the intersection/roadway they control" [0002-0003].
Claims 27-29 are rejected under 35 U.S.C. 103 as being unpatentable over Ofek in view of Kadar and Matsunaga.
Regarding claim 27, Ofek and Kadar teach the system of claim 26. Ofek does not explicitly teach wherein the processor is configured to label the region of interest and associating objects detected within the region of interest with the label of the region of interest.
Matsunaga, in the same field of endeavor of vehicle ROI measurement, teaches wherein the processor is configured to label the region of interest and associating objects detected within the region of interest with the label of the region of interest ([Abstract] (A) is map information representing reachable ranges for a moving body such as a vehicle. The map information is divided into a plurality of regions. Regions that are filled in represent reachable ranges for the moving body. In (A), there are three linked region groups representing reachable ranges for the moving body. (B) shows a state in which a label number has been assigned to each linked region group. [pg. 11 para. 2] The detecting unit 406 detects a specific area group related to the position of the moving body from among a plurality of area groups indicating the reachable range of each moving body on the map information. Specifically, the detection unit 406 detects a specific connected region group related to the position of the moving body from among a plurality of connected region groups on the map information. The specific connected area group is, for example, a connected area group including the own vehicle position or a connected area group closest to the own vehicle position. [pg. 3 para. 4] In the image processing example 1 of FIG. 1, the position of the moving body (own vehicle position) exists in the connected region group R3).
PNG
media_image5.png
560
1227
media_image5.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ofek with the teachings of Matsunaga to label the region of interest and associate objects detected within the region of interest with the label of the region of interest because "the image processing apparatus selects and deletes areas unnecessary for display from a reachable range that is divided into a plurality of parts, and has a predetermined number, area, perimeter, and shape… In particular, the image processing apparatus sorts out a specific area in consideration of the vehicle position" [pg. 3 para. 2].
Regarding claim 28, Ofek and Kadar teach the system of claim 26. Ofek further teaches the panorama.
PNG
media_image6.png
441
506
media_image6.png
Greyscale
Ofek does not explicitly teach wherein the processor is configured to: associate pixels in the image with latitude and longitude coordinates; and associate a detected object with latitude and longitude coordinates of the pixels that include the detected object.
Matsunaga, in the same field of endeavor of vehicle ROI measurement, teaches wherein the processor is configured to: associate pixels in the image with latitude and longitude coordinates ([pg. 3 para. 3] FIG. 1 is an explanatory diagram of an image processing example 1 according to the first embodiment. (A) is the map information which shows the reachable range of moving bodies, such as a vehicle. The map information is divided into a plurality of areas. Each region corresponds to one or more pixels, for example. [pg. 17 para. 6-7] The dividing unit 404 according to the present embodiment divides the map data stored in the storage device…As shown in FIG. 11, the dividing unit 404 first generates longitude / latitude information (x, y) having a point group 1100 in absolute coordinates based on the longitude x and latitude y of each of a plurality of reachable points. . The origin (0, 0) of the longitude / latitude information (x, y) is at the lower left of FIG. Then, the dividing unit 404 calculates distances w1 and w2 from the longitude ofx of the current point 500 of the vehicle to the maximum longitude x_max and the minimum longitude x_min of the reachable point farthest in the longitude x direction. Further, the dividing unit 404 calculates the distances w3 and w4 from the latitude of the current point 500 of the vehicle to the maximum latitude y_max and the minimum latitude y_min of the reachable point farthest in the latitude y direction);
and associate a detected object with latitude and longitude coordinates of the pixels that include the detected object ([pg. 3 para. 3-4] FIG. 1 is an explanatory diagram of an image processing example 1 according to the first embodiment. (A) is the map information which shows the reachable range of moving bodies, such as a vehicle. The map information is divided into a plurality of areas. Each region corresponds to one or more pixels, for example. In (A), the filled area indicates the reachable range of the moving object. A region formed by connecting filled regions is referred to as a connected region group. In (A), there are three connected region groups indicating the reachable range of the moving object. (B) is the next state of (A) and shows a state where a label number is assigned to each connected region group… In the image processing example 1 of FIG. 1, the position of the moving body (own vehicle position) exists in the connected region group R3. [pg. 11 para. 2] The detecting unit 406 detects a specific area group related to the position of the moving body from among a plurality of area groups indicating the reachable range of each moving body on the map information. Specifically, the detection unit 406 detects a specific connected region group related to the position of the moving body from among a plurality of connected region groups on the map information. The specific connected area group is, for example, a connected area group including the own vehicle position or a connected area group closest to the own vehicle position. [pg. 6 para. 1-4] The camera 314 captures images inside or outside the vehicle. The image may be either a still image or a moving image. For example, the outside of the vehicle is photographed by the camera 314, and the photographed image is analyzed by the CPU 301…The GPS unit 316 receives radio waves from GPS satellites and outputs information indicating the current position of the vehicle. The output information of the GPS unit 316 is used when the CPU 301 calculates the current position of the vehicle together with output values of various sensors 317 described later. The information indicating the current position is information for specifying one point on the map data, such as latitude / longitude and altitude).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ofek with the teachings of Matsunaga associate detected objects in the image with latitude and longitude coordinates associated with pixels because " the display control unit 410 may extract the reachable range of the mobile body based on the longitude / latitude information of the area to which the reachable identification information is given, and display the reachable range on the display unit 411" [pg. 12 para. 7] and "the image processing apparatus selects and deletes areas unnecessary for display from a reachable range that is divided into a plurality of parts, and has a predetermined number, area, perimeter, and shape… In particular, the image processing apparatus sorts out a specific area in consideration of the vehicle position" [pg. 3 para. 2].
Regarding claim 29, Ofek, Kadar, and Matsunaga teach the system of claim 28. Ofek does not explicitly teach wherein the processor is configured to compare the latitude and longitude coordinates of the detected object in at least two time-spaced images to measure the speed of the detected object.
Matsunaga, in the same field of endeavor of vehicle ROI measurement, teaches wherein the processor is configured to compare the latitude and longitude coordinates of the detected object in at least two time-spaced images to measure the speed of the detected object ([pg. 6 para. 1-4] The camera 314 captures images inside or outside the vehicle. The image may be either a still image or a moving image. For example, the outside of the vehicle is photographed by the camera 314, and the photographed image is analyzed by the CPU 301…The GPS unit 316 receives radio waves from GPS satellites and outputs information indicating the current position of the vehicle. The output information of the GPS unit 316 is used when the CPU 301 calculates the current position of the vehicle together with output values of various sensors 317 described later. The information indicating the current position is information for specifying one point on the map data, such as latitude / longitude and altitude. Various sensors 317 output information for determining the position and behavior of the vehicle, such as a vehicle speed sensor, an acceleration sensor, an angular velocity sensor, and a tilt sensor. The output values of the various sensors 317 are used by the CPU 301 to calculate the current position of the vehicle and the amount of change in speed and direction).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ofek with the teachings of Matsunaga to compare latitude and longitude coordinates to measure the speed so that "the calculation unit 402 estimates an estimated energy consumption amount in a predetermined section based on information about the speed of the moving body and the moving body information" [pg. 7 para. 3].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571) 272-4077. The examiner can normally be reached M-F 9-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights