DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 14, and 15 are rejected under 35 U.S.C. 103 as being unpatenable by Morisawa; Keisuke et al. (US 20200175702 A1; hereinafter simply referred to as Morisawa) in view of Konno; Megumi et al. (JP 2020112928 A; hereinafter simply referred to as Konno Megumi; translated via ESPACENET)
Regarding independent claim 1, Morisawa teaches:
An information processing apparatus comprising: one or more hardware processors; and one or more memories storing one or more programs configured to be executed by the one or more hardware processors (See ¶ 5, 39, 45, and 160 and figure 3B wherein an information/image processing apparatus comprises a processor, 351 in figure 3B, and a memory, 353 figure 3B, storing instructions/programs configured to be executed by the processor)
obtaining foreground information indicating a region corresponding to a foreground object in each of captured images obtained by causing a plurality of image capturing devices (camera 101a – 101r in figure 3A, also see ¶ 39 and 46) to perform image capturing, respectively, the plurality of image capturing devices configured to capture images of an image capturing region from directions varying from one another; (See ¶ 32, 34, 39, 46 claim 1, and Figure 1a – 1c, 2 and 3A wherein foreground information indicating a region (object area) corresponding to a foreground object is obtained using a plurality of cameras wherein the images from the plurality of cameras capture the object region from multiple views/angles (varying directions)
wherein in the case where the distance information is unobtainable by a first obtaining method, the distance information is obtained by a second obtaining method, and at least one of the first obtaining method and the second obtaining method uses at least part of the plurality of image capturing devices as a stereo camera to obtain the distance information. (See ¶ 63, 66-68, 74-81, 92, 37, 38, Claims 2, 3, and 5 and figures 1C, 2, 9C and 10 wherein when the image indicating the foreground area is improper (distance information for image is unobtainable as the foreground area is incorrect, therefore first method fails) a second method is performed being the processing for inhibiting the negative influence of the improper foreground mask image, wherein as seen in figure 1C and 2, a stereo vision camera setup is used).
Morisawa does not explicitly disclose obtaining distance information indicating a distance from a reference point to the foreground object; and obtaining three-dimensional shape data indicating a three-dimensional shape of the foreground object based on the foreground information and the distance information.
However, Konno teaches of obtaining distance information indicating a distance from a reference point to the foreground object, (See ¶ 43 and 134 and figure 13 wherein depth is calculated which refers to the distance (distance information) in the depth direction from the optical center of the camera (reference point), 5A in figure 13, to the foreground object)
and obtaining three-dimensional shape data indicating a three-dimensional shape of the foreground object based on the foreground information and the distance information. (See ¶ 113 – 115, 137 – 139, 3 and figure 12 wherein a three-dimensional shape data indicating a three-dimensional shape of the foreground object is obtained, being the three dimensional foreground model obtained via the separation unit, 23 in figure 12, and the foreground generation unit, 15E in figure 12, which use the silhouette image (foreground information) and depth image (distance information) to obtain the model).
As taught by Konno using foreground information and distance information to create the three-dimensional shape data allows for the separation of foreground and background to be completed even when the colors of the foreground and background are similar. (See ¶ 115 wherein the foreground and background are still able to be separated even when they comprise similar colors due to the use of distance and foreground information). As both the teachings of Morisawa and Konno deal with the technical field of image processing regarding the foreground and background of an image, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Morisawa and Konno to teach of obtaining three-dimensional shape data indicating a three-dimensional shape of the foreground object based on the foreground information and the distance information in order for foreground and background separation to still occur even when they comprise similar colors.
Regarding independent claim 14, claim 14 is a method claim corresponding to claim 1. Please see the discussion of claim 1 above.
Regarding independent claim 15, claim 15 is a non-transitory computer readable storage medium claim corresponding to claim 1. Please see the discussion of claim 1 above. Furthermore, Morisawa teaches of a non-transitory computer readable storage medium comprising a program to perform a method to obtain three-dimensional shape data of an object. (See ¶ 160 and 5 wherein a non-transitory computer readable storage medium comprises instructions/programs for to perform a method to obtain three-dimensional shape data of an object).
Claims 2 – 5 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Morisawa; Keisuke et al. (US 20200175702 A1; hereinafter simply referred to as Morisawa) in view of Konno; Megumi et al. (JP 2020112928 A; hereinafter simply referred to as Konno Megumi; translated via ESPACENET) and further in view of Handa; Masahiro et al. (US 20190356906 A1; hereinafter simply referred to as Handa).
Regarding dependent claim 2, Morisawa in view of Konno teaches:
In the case where the distance information is unobtainable by the first obtaining method, the distance information is obtained by the second obtaining method decided. (See ¶ 63, 66-68, 74-81, 92, 37, 38, Claims 2, 3, and 5 and 9C and 10 wherein when the image indicating the foreground area is improper (distance information for image is unobtainable as the foreground area is incorrect, therefore first method fails) a second method is performed being the processing for inhibiting the negative influence of the improper foreground mask image).
Morisawa does not explicitly disclose deciding the second obtaining method such that a region in which the distance information is unobtainable is reduced, in the case where the distance information is unobtainable by the first obtaining method.
However, Handa teaches of deciding the second obtaining method such that a region in which the distance information is unobtainable is reduced, (See ¶ 371 - 373 wherein the partial region 3401 is created which is a smaller region (reduced region) where the foreground can be extracted).
As taught by Handa the reduction of the region allows for higher resolution. (See ¶ 371 wherein the reduction of the background region allows for the system to cope with higher resolutions). As both the teachings of Morisawa in view of Konno and Handa deal with the technical field of image processing for 3D model generation it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Morisawa in view of Konno with Handa to teach of deciding the second obtaining method such that a region in which the distance information is unobtainable is reduced in order for a system to deal with higher resolutions.
Regarding dependent claim 3, Morisawa in view of Konno and Handa teaches:
The first obtaining method includes a method of obtaining the distance information by using at least part of the plurality of image capturing devices as the stereo camera, and the distance information is obtained by the second obtaining method in the case where the distance information is unobtainable by the first obtaining method using the stereo camera. (See Morisawa ¶ 63, 66-68, 74-81, 92, 37, 38, Claims 2, 3, and 5 and figures 1C, 2, 9C and 10 wherein when the image indicating the foreground area is improper (distance information for image is unobtainable as the foreground area is incorrect, therefore first method fails) a second method is performed being the processing for inhibiting the negative influence of the improper foreground mask image, wherein as seen in figure 1C and 2, a stereo vision camera setup is used for the first method).
Regarding dependent claim 4, Morisawa in view of Konno and Handa teaches:
The distance information is obtained by the second obtaining method in the case where the distance information is unobtainable by the first obtaining method using the stereo camera due to an abnormality in transmission of the foreground information. (See Morisawa ¶ 37 and 38 wherein the second method is used due to an abnormality in transmission (abnormality in transmission system that causes the foreground area to not correctly represent the silhouette of an object) of the image data (foreground information)).
Regarding dependent claim 5, Morisawa in view of Konno and Handa teaches:
The second obtaining method is decided based on abnormality information indicating the abnormality. (See Morisawa ¶ 37 and 38 wherein the second method is only chosen based on the abnormality occurring, as when the abnormality occurs leading to improper foreground information, the second method is implemented based on that abnormality).
Regarding dependent claim 13, Morisawa in view of Konno does not explicitly disclose:
The three-dimensional shape data is obtained by generating temporary three-dimensional shape data indicating the three-dimensional shape of the foreground object based on the foreground information and correcting the temporary three-dimensional shape data based on the distance information.
However Handa teaches of the three-dimensional shape data is obtained by generating temporary three-dimensional shape data indicating the three-dimensional shape of the foreground object based on the foreground information and correcting the temporary three-dimensional shape data based on the distance information. (See Handa ¶ 152 and 7 wherein a virtual viewpoint image is continuously corrected/updated (creating a temporary shape data that is corrected) based on new image/distance information wherein the virtual viewpoint image is created using model based rendering (MBR) which creates the three-dimensional shape data of the foreground object in the virtual viewpoint image).
As taught by Handa the three-dimensional shape data being generated temporally and then corrected based on the distance information allows for the virtual viewpoint image to be updated based on the new image data received. (See ¶ wherein the virtual viewpoint image is updated based on the new received distance information from the updated image data). As both the teachings of Morisawa in view of Konno and Handa deal with the technical field of image processing for 3D model generation it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Morisawa in view of Konno with Handa to teach of the three-dimensional shape data is obtained by generating temporary three-dimensional shape data indicating the three-dimensional shape of the foreground object based on the foreground information and correcting the temporary three-dimensional shape data based on the distance information in order for the virtual viewpoint image to be updated based on the new image data received.
Claims 6 – 12 are rejected under 35 U.S.C. 103 as being unpatentable over Morisawa; Keisuke et al. (US 20200175702 A1; hereinafter simply referred to as Morisawa) in view of Konno; Megumi et al. (JP 2020112928 A; hereinafter simply referred to as Konno Megumi; translated via ESPACENET) in view of Handa; Masahiro et al. (US 20190356906 A1; hereinafter simply referred to as Handa) further in view of Sugio; Toshiyasu et al. (US 20200226794 A1; hereinafter simply referred to as Sugio)
Regarding dependent claim 6, Morisawa in view of Konno and Handa does not explicitly disclose:
The second obtaining method includes a method of obtaining the distance information by using at least part of the plurality of image capturing devices as the stereo camera, and the image capturing devices to be used as the stereo camera in the second obtaining method are decided from among the plurality of image capturing devices based on the abnormality information.
However, Sugio teaches of the second obtaining method includes a method of obtaining the distance information by using at least part of the plurality of image capturing devices as the stereo camera, and the image capturing devices to be used as the stereo camera in the second obtaining method are decided from among the plurality of image capturing devices based on the abnormality information. (See ¶ 177, 611 – 614, 407 wherein an alternative sensor/camera is used in the second obtaining method (used in a case wherein a first sensor/camera fails) based on the abnormality information (generated three dimensional data not meeting a threshold) wherein the second method uses a stereo camera from among a plurality of combinations of image capturing devices).
As taught by Sugio the use of an alternative emergency camera/sensor in the case of a failing sensor/camera allows for a vehicle to continue to operate after the first sensor/camera malfunctions. (See ¶ 611 – 615 wherein an alternative sensor/camera is used when the first camera/sensor malfunctions allowing for the vehicle to continue to operate at full capacity). As both the teachings of Morisawa in view of Konno and Handa and Sugio deal with the technical field of image processing regarding three-dimensional data, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Morisawa in view of Konno and Handa with Sugio to teach of the second obtaining method includes a method of obtaining the distance information by using at least part of the plurality of image capturing devices as the stereo camera, and the image capturing devices to be used as the stereo camera in the second obtaining method are decided from among the plurality of image capturing devices based on the abnormality information in order for a vehicle/system to continue to operate with the use of alternate cameras/sensors in case of a camera/sensor failure.
Regarding dependent claim 7, Morisawa in view of Konno and Handa does not explicitly disclose:
The second obtaining method includes a method of obtaining the distance information based on output from a range sensor, and the range sensor to be used in the second obtaining method is decided based on the abnormality information.
However, Sugio teaches of the second obtaining method includes a method of obtaining the distance information based on output from a range sensor, and the range sensor to be used in the second obtaining method is decided based on the abnormality information. (See ¶ 177, 611 – 614, 3, 407, 455, 542 wherein an alternative sensor/camera is used in the second obtaining method (used in a case wherein a first sensor/camera fails) based on the abnormality information (generated three dimensional data not meeting a threshold), wherein the alternative sensor is a rangefinder (range sensor)).
As taught by Sugio the use of an alternative emergency camera/sensor (range sensor) in the case of a failing sensor/camera allows for a vehicle to continue to operate after the first sensor/camera malfunctions. (See ¶ 611 – 615 wherein an alternative sensor/camera is used when the first camera/sensor malfunctions allowing for the vehicle to continue to operate at full capacity). As both the teachings of Morisawa in view of Konno and Handa and Sugio deal with the technical field of image processing regarding three-dimensional data, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Morisawa in view of Konno and Handa with Sugio to teach of the second obtaining method includes a method of obtaining the distance information based on output from a range sensor, and the range sensor to be used in the second obtaining method is decided based on the abnormality information in order for a vehicle/system to continue to operate with the use of alternate cameras/sensors (range sensor) in case of a camera/sensor failure.
Regarding dependent claim 8, Morisawa in view of Konno and Handa teaches:
The distance information is obtained by the second obtaining method in the case where the distance information is unobtainable by the first obtaining method. (See ¶ 37, 38, Claims 2, 3, and 5 wherein when the image indicating the foreground area is improper (distance information for image is unobtainable as the foreground area is incorrect, therefore first method fails) a second method is performed being the processing for inhibiting the negative influence of the improper foreground mask image).
Morisawa in view of Konno and Handa does not explicitly disclose the first obtaining method includes a method of obtaining the distance information based on output from a range sensor.
However, Sugio teaches of the first obtaining method includes a method of obtaining the distance information based on output from a range sensor (See ¶ 3, 407, 455, 542, 611 – 614, wherein distance information is obtained via a rangefinder (range sensor) wherein there are a plurality of sensor/camera options that can be alternated for different cases such as camera/sensor failure).
As taught by Sugio the use of a rangefinder/range sensor to obtain distance information allows for a self-location estimation by a vehicle to be implemented. (See ¶ 542 wherein the use of a rangefinder allows for the implementation of self-location estimation by a vehicle). As both the teachings of Morisawa in view of Konno and Handa and Sugio deal with the technical field of image processing regarding three-dimensional data, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Morisawa in view of Konno and Handa with Sugio to teach of the first obtaining method includes a method of obtaining the distance information based on output from a range sensor in order to implement self-location estimation by a vehicle.
Regarding dependent claim 9, Morisawa in view of Konno and Handa and Sugio teaches:
The distance information is obtained by the second obtaining method in the case where the distance information is unobtainable by the first obtaining method using the range sensor due to a sensor abnormality in the distance information based on the output from the range sensor. (See Sugio ¶ 611, 612, 613, 542, 606 wherein when the rangefinder fails to meet a reference value threshold for accuracy (abnormality in distance information) a second obtaining method being an alternate/emergency sensor/camera is used.)
Regarding dependent claim 10, Morisawa in view of Konno, Handa and Sugio teaches:
The second obtaining method is decided based on sensor abnormality information indicating the sensor abnormality. (See Sugio ¶ 611, 612, 613, 542, 606 wherein when the rangefinder fails to meet a reference value threshold for accuracy (abnormality in distance information indicating abnormality in the sensor) a second obtaining method being an alternate/emergency sensor/camera is used.)
Regarding dependent claim 11, Morisawa in view of Konno, Handa and Sugio teaches:
The second obtaining method includes a method of obtaining the distance information by using at least part of the plurality of image capturing devices as the stereo camera, and the image capturing devices to be used as the stereo camera in the second obtaining method are decided from among the plurality of image capturing devices based on the abnormality information. (See Sugio ¶ 177, 611 – 614, 3, 542, 613, 648, 407 wherein an alternative sensor/camera is used in the second obtaining method (used in a case wherein a first sensor/camera fails) based on the abnormality information (generated three dimensional data not meeting a threshold) wherein the first method uses a rangefinder (range sensor in the form of LiDAR) and the second method uses a stereo camera from among a plurality of combinations of image capturing devices).
Regarding dependent claim 12, Morisawa in view of Konno, Handa and Sugio teaches:
The range sensor is a sensor configured to measure a distance to an object by observing reflected light of emitted light. (See Sugio ¶ 542, 613, 648 wherein the range sensor is a sensor configured to measure a distance to an object by observing reflected light of emitted light represented by LiDAR sensors).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See attached PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEJANDRO HERNANDEZ whose telephone number is (703)756-1876. The examiner can normally be reached M-F 8 am - 5 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEJANDRO HERNANDEZ/Examiner, Art Unit 2661
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661