DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-12 & 19-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Satoshi Yoshikawa Et. Al. (Pat. Pub. WO-2021200432-A1, herein after “Yoshikawa”).
In regard to claims 1, 19, & 20, Yoshikawa teaches [a]n information processing method comprising:
performing, based on sensing information acquired in a moving body “The image pickup unit 311 is an image pickup device such as a camera, and acquires an image (moving image)” (Yoshikawa, Page 15) where the image pickup unit is movable and has sensors “an instruction may be given to a moving body equipped with a camera such as a robot or a drone” (Yoshikawa, Page 13), processing of detecting a defective subject that may degrade a performance of three- dimensional modeling of a subject of interest using images shot from a plurality of viewpoints “a UI (user interface) or the like that detects a region where three dimensional reconstruction (generation of a three-dimensional model) is difficult and instructs a shooting position or a shooting posture based on the detection result” (Yoshikawa, Page 5) which describes a region where three-dimensional reconstruction is difficult, and produces instructions of a position for the camera to overcome the trouble in reconstruction; and
outputting, when the defective subject is detected, video shooting plan information for video shooting to reduce a region of the defective subject included in the shot images “FIG. 25 is a plan view showing a state of photography in the target space” (Yoshikawa, Page 18) where a shooting plan can be generated to film a desired area if the desired area is not visible, for example if the desired area is “hidden by another object” (Yoshikawa, Page 15).
PNG
media_image1.png
228
341
media_image1.png
Greyscale
Yoshikawa, Fig. 25, depicts two cameras (items A and B) which have paths (items C and D, respectively) where the cameras are able to follow the generated path to accurately film the desired area.
In regard to claim 19, claim 1 is substantially similar to claim 19, hence the rejection analysis for claim 1 is also applied to claim 19. teach the additional limitations of [a]n information processing device comprising:
a subject detection unit that performs “The terminal device 100 includes an imaging unit 101, a control unit 102, a position / orientation estimation unit 103, a three-dimensional reconstruction unit 104, an image analysis unit 105” (Yoshikawa, Page 6) where a desired object can be imaged or detected, based on sensing information acquired in a moving body (Yoshikawa, Pages 13 & 15), processing of detecting a defective subject that may degrade a performance of three-dimensional modeling of a subject of interest using images shot from a plurality of viewpoints (Yoshikawa, Pages 5, 13, & 15); and
a video shooting planning unit that outputs, when the defective subject is detected, video shooting plan information for video shooting to reduce a region of the defective subject included in the shot images (Yoshikawa, Pages 15 & 18).
In regard to claim 20, claim 1 is substantially similar to claim 20, hence the rejection analysis for claim 1 is also applied to claim 20. teach the additional limitations of [a] program for causing a computer to execute processing of:
performing, based on sensing information acquired in a moving body (Yoshikawa, Page 13 & 15), processing of detecting a defective subject that may degrade a performance of three- dimensional modeling of a subject of interest using images shot from a plurality of viewpoints (Yoshikawa, Page 5); and
outputting, when the defective subject is detected, video shooting plan information for video shooting to reduce a region of the defective subject included in the shot images (Yoshikawa, Pages 15 & 18).
In regard to claim 2, Yoshikawa teaches [t]he information processing method according to claim 1, wherein the video shooting plan information includes a pathway of the moving body, a position and orientation of the moving body, and an orientation of a camera included in the moving body “FIG. 35 is a plan view of a three-dimensional map, and shows a photographed area and a low-precision area. Further, the current camera position 331 (position and direction of the photographing device 301), the current route 332, and the past route 333 are shown” (Yoshikawa, Page 21) where the camera route to be followed includes the camera position and shot direction, which is read as orientation. The plan is generated to film or scan an object based on any obstructions.
In regard to claim 3, Yoshikawa teaches [t]he information processing method according to claim 2, wherein, when the defective subject is detected, the video shooting plan information in which at least one of the position and orientation of the moving body and the orientation of the camera has been corrected is output “the UI unit 108 determines whether or not the shooting position candidate or the estimation failure signal has been received (S603). Here, the estimation failure signal is a signal transmitted from the position / attitude estimation unit 103 when the position / attitude estimation in the position / attitude estimation unit 103 fails” (Yoshikawa, Page 12), additionally, “When the UI unit 108 receives the shooting position candidate (Yes in S603), it displays that there is a shooting position candidate (S604) and presents the shooting position candidate (S605)” (Yoshikawa, Page 12) where a defective subject causes the estimation failure signal to be sent, and a new shooting candidate will be sent to the user.
In regard to claim 4, Yoshikawa teaches [t]he information processing method according to claim 3, comprising correcting at least one of the position and orientation of the moving body and the orientation of the camera so that the subject of interest is included within an angle of view of the shot image, and the defective subject is hidden by the subject of interest in the shot image or the defective subject is excluded from the angle of view of the shot image “the point cloud analysis unit 106 detects a region of the point cloud generated by using the peripheral region of the image in which lens distortion or the like is likely to occur, and determines a shooting position candidate such that the region is captured in the center of the camera. .. The determined shooting position candidate is output to the UI unit 108 and presented to the user” (Yoshikawa, Page 7) where distortion may be detected in the peripheral of the camera for example, which prompts the moving body to change its position so that the object is centered and the distortion is excluded from the shot.
In regard to claim 5, Yoshikawa teaches [t]he information processing method according to claim 3, wherein the processing of detecting the defective subject is performed each time the shot image is shot “In FIG. 2, the region shown in light ink indicates that the imaging unit 101 is continuously photographing. In order to generate a high-quality three-dimensional model, the terminal device 100 analyzes the image and the shooting position and posture in real time, and gives a shooting instruction to the user. Here, the real-time analysis is to perform the analysis while taking a picture” (Yoshikawa, Page 7) where the device takes photos as it moves, and scans the object for areas that may be difficult to reconstruct, and
the video shooting plan information is output each time the defective subject is detected “The terminal device 100 predicts a shooting position and posture that can secure parallax that makes it easy to restore the area, and presents the predicted shooting position and posture on the UI” (Yoshikawa, Page 7) where the predicted shooting position and posture are output as defects are spotted.
In regard to claim 6, Yoshikawa teaches [t]he information processing method according to claim 3, wherein the processing of detecting the defective subject is performed based on the shot images from the plurality of viewpoints obtained by executing video shooting a plurality of times based on the video shooting plan information “The position / orientation integration unit 313 integrates the position / orientation of the camera estimated in each shooting when performing multiple shootings in one environment, and calculates the position / orientation that can be handled in the same space” (Yoshikawa, Page 16) where multiple camera shootings are performed on the same environment and can follow the same path, and
the information processing method comprises generating, when the defective subject is detected from any of the shot images from the plurality of viewpoints, the video shooting plan information for re-shooting the shot image in which the defective subject has been detected “Specifically, the area detection unit 314 generates three-dimensional position information (three-dimensional point cloud, three-dimensional model, depth image, etc.) using the estimation result of the position and orientation and the image, and the generated three-dimensional position. The information is used to detect areas where 3D reconstruction is not possible or areas where 3D reconstruction is inaccurate” (Yoshikawa, Page 17) where information is generated based on the camera position and orientation when capturing the object image, including the defective area, and shooting may be repeated as needed to resolve the defective area via a shooting plan.
In regard to claim 7, Yoshikawa teaches [t]he information processing method according to claim 3, comprising reflecting, when the defective subject is detected, evaluation information regarding a possibility that the defective subject will degrade the performance of the three-dimensional modeling in a map for setting a pathway of the moving body “the terminal device 100 estimates the position and orientation of the camera during shooting, and determines a region that is difficult to restore based on the estimation result and the shot image” (Yoshikawa, Page 7) where the broadest reasonable interpretation of evaluation information is read as the location, or position and orientation of the moving body as the defective subject that would be difficult to restore is observed in real time. Additionally, “the UI unit 108 may display shooting position candidates on map information (two-dimensional or three-dimensional)” (Yoshikawa, Page 13) map information is taught; and
generating the video shooting plan information based on the map in which the evaluation information has been reflected “the moving body may move to the determined shooting position candidate and perform shooting. According to this, a highly accurate three-dimensional model can be stably generated even in an automatically controlled device” (Yoshikawa, Page 13) where the moving body may move automatically to given instruction, such as a pathway, and the movement to the shooting path is read as a shooting plan.
In regard to claim 8, Yoshikawa teaches [t]he information processing method according to claim 7, wherein, when the detected defective subject cannot be excluded from the shot image, the evaluation information is reflected in the map “the photographing device 301 may determine that the region where the depth value could not be calculated from the parallax image cannot be restored (not photographed)” (Yoshikawa, Page 23) where there can be an instance where the image cannot be restored “Further, the area detection unit 314 stores the information of the detected area in the area information storage unit 319. The area detection unit 314 may detect an area capable of three-dimensional reconstruction and determine that the area other than the area cannot be three-dimensionally reconstructed” (Yoshikawa, Page 16).
In regard to claim 9, Yoshikawa teaches [t]he information processing method according to claim 7, wherein the evaluation information includes information that indicates a position of the moving body at a time when the shot image in which the defective subject is detected was shot “the terminal device 100 estimates the position and orientation of the camera during shooting, and determines a region that is difficult to restore based on the estimation result and the shot image” (Yoshikawa, Page 7) where the evaluation information is read as the location, or position and orientation of the moving body as the defective subject that would be difficult to restore is observed in real time.
In regard to claim 10, Yoshikawa teaches [t]he information processing method according to claim 1, comprising performing, after executing video shooting a plurality of times based on the video shooting plan information, three-dimensional modeling of the subject of interest using the shot images from the plurality of viewpoints “Next, the three-dimensional reconstruction unit 104 generates a three-dimensional model by performing three-dimensional reconstruction using the acquired plurality of images and the plurality of position / orientation information (S503)” (Yoshikawa, Page 11) where the plurality of images are obtained by the camera which follows a shooting plan to create a three-dimensional model.
In regard to claim 11, Yoshikawa teaches [t]he information processing method according to claim 1, comprising performing three-dimensional modeling of the subject of interest using the shot images “the three-dimensional reconstruction unit 104 acquires a plurality of images or videos from the video storage unit 111 (S501)” (Yoshikawa, Page 11) where shot images or videos can be used for three-dimensional modeling in parallel with executing video shooting a plurality of times based on the video shooting plan information “the imaging unit 101 may perform streaming transmission that appropriately transmits an image at the same time as shooting, or may collectively transmit an image at that time at regular intervals. That is, the image information is one or a plurality of images (frames) included in the video” (Yoshikawa, Page 7) where video shooting may be done as the modeling utilizes shot images, and “FIG. 35 is a plan view of a three-dimensional map, and shows a photographed area and a low-precision area. Further, the current camera position 331 (position and direction of the photographing device 301), the current route 332, and the past route 333 are shown” (Yoshikawa, Page 21) where a shooting plan is used to direct the moving body.
In regard to claim 12, Yoshikawa teaches [t]he information processing method according to claim 1, wherein the processing of detecting the defective subject is performed based on attribute information of an object obtained by semantics estimation using the sensing information “labels such as window frames, desks, and walls are added to each pixel by a method such as Semantic Segmentation on an image, and the target pixels are collectively selected by selecting the labels” (Yoshikawa, Page 9) where semantic segmentation can be applied to the imaging techniques used.
PNG
media_image2.png
622
440
media_image2.png
Greyscale
Yoshikawa, Figs. 5 & 6, depicting a location with an object that is to be recognized. Semantic segmentation may apply labels to objects, or a user may designate an arbitrary area (as outlined in Fig. 6).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshikawa in view of Travis Schoyck Et. Al. (Pat. Pub. US-20190068829-A1, herein after “Schoyck”).
In regard to claim 13, Yoshikawa teaches [t]he information processing method according to claim 1, wherein the processing of detecting the defective subject is performed based on movement information of an object obtained by optical flow estimation using the sensing information.
Yoshikawa fails to explicitly teach wherein the processing of detecting the defective subject is performed based on movement information of an object obtained by optical flow estimation using the sensing information.
Schoyck teaches wherein the processing of detecting the defective subject is performed based on movement information of an object obtained by optical flow estimation using the sensing information “the VIO module 316 may combine visual information, such as optical flow or feature tracking information, with inertial information, such as information from an accelerometer or gyroscope. The VIO [visual inertial odometry] module 316 may also combine distance and ground information, such as ultrasound range measurements, or 3D depth or disparity data” (Schoyck, ¶ [0066]) where the use of optical flow helps detect the motion of objects and pixels, alongside the use of many other listed features.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of detecting a defective subject in an image taught by Yoshikawa with the method of optical flow to detect motion of objects within images taught by Schoyck to detect a moving object within a set of images or video and determine if the subject is a defect. The suggestion/motivation to do so would have been to only scan an intended object while capturing images or video.
In regard to claim 14, Yoshikawa teaches [t]he information processing method according to claim 1, wherein the processing of detecting the defective subject is performed based on a surface shape of an object obtained by shape recognition using the sensing information.
Yoshikawa fails to explicitly teach wherein the processing of detecting the defective subject is performed based on a surface shape of an object obtained by shape recognition using the sensing information.
Schoyck teaches wherein the processing of detecting the defective subject is performed based on a surface shape of an object obtained by shape recognition using the sensing information “ the obstruction may be further classified as a defect or as a vision-blocking structure based, for example, on properties of the identified region (e.g., type of shape, line, shadow, etc. in the image data)” (Schoyck, ¶ [0076]) where an irregular shape in the image may be determined to be a defect.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of detecting a defective subject in an image taught by Yoshikawa with the method of detecting irregular shapes taught by Schoyck to detect an irregular shape and determine if the subject is a defect. The suggestion/motivation to do so would have been to only scan an intended object while capturing images or video.
In regard to claim 15, Yoshikawa teaches [t]he information processing method according to claim 1, wherein the processing of detecting the defective subject is performed based on normal information of an object surface obtained by calculating a surface normal using the sensing information.
Yoshikawa fails to teach wherein the processing of detecting the defective subject is performed based on normal information of an object surface obtained by calculating a surface normal using the sensing information.
Schoyck teaches wherein the processing of detecting the defective subject is performed based on normal information of an object surface obtained by calculating a surface normal using the sensing information “Feature tracking between images or frames may be improved in various embodiments by estimating a surface normal in a manner that accounts for appearance transformation between views” (Schoyck, ¶ [0072]) where estimating a feature normal is taught, and may be used to track objects and determine obstructions/defects within an image.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of detecting a defective subject in an image taught by Yoshikawa with the method of feature tracking by estimating a surface normal taught by Schoyck to utilize a surface normal in detecting defective subjects in an image. The suggestion/motivation to do so would have been to only scan an intended object while capturing images or video.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Yoshikawa in view of Jordan Holt Et. Al. (Pat. Pub. US-20170313439-A1, herein after “Holt”).
In regard to claim 16, Yoshikawa teaches [t]he information processing method according to claim 1, wherein the processing of detecting the defective subject is performed based on texture information of an object obtained by pattern matching using the sensing information.
Yoshikawa fails to teach wherein the processing of detecting the defective subject is performed based on texture information of an object obtained by pattern matching using the sensing information.
Holt teaches wherein the processing of detecting the defective subject is performed based on texture information of an object obtained by pattern matching using the sensing information “the video feed including the target landing area is processed using an obstruction detection algorithm. Color histogram anomaly is the preferred obstruction detection algorithm; ... Other examples may include motion or moving object detection, texture anomaly detection …” (Holt, ¶ [0033]) where texture information is used to detect an anomaly.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of detecting a defective subject in an image taught by Yoshikawa with the method of texture detection taught by Holt to detect a defective area based on differences in texture. The suggestion/motivation to do so would have been to only scan an intended object while capturing images or video.
Claim(s) 17 & 18 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshikawa in view of Totsuka Atsushi (Pat. Pub. WO-2022249562-A1, herein after “Atsushi”).
In regard to claim 17, Yoshikawa teaches [t]he information processing method according to claim 1, wherein the processing of detecting the defective subject is performed with at least one of a blown-out highlight and a crushed shadow included in the sensing information as the defective subject.
Yoshikawa fails to teach wherein the processing of detecting the defective subject is performed with at least one of a blown-out highlight and a crushed shadow included in the sensing information as the defective subject.
Atsushi teaches wherein the processing of detecting the defective subject is performed with at least one of a blown-out highlight and a crushed shadow included in the sensing information as the defective subject “as a warning display to the user, for example, a display prompting the user to manually change the exposure setting (exposure setting) so as to obtain an appropriate exposure time that does not cause blown-out highlights or blocked-up shadows is displayed” (Atsushi, Page 10) where a warning display can be output upon detection of blown-out highlights or shadows and prompt for user correction of exposure settings.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of defect detection in an image taught by Yoshikawa with the method of detecting blown-out highlights and shadows taught by Atsushi to detect blown-out highlights or excessive shadows as defects. The suggestion/motivation to do so would have been to more accurately detect the image which is intended to be scanned.
In regard to claim 18, Yoshikawa teaches [t]he information processing method according to claim 1, wherein the sensing information includes at least one of an RGB image “The image pickup unit 311 may be a device capable of capturing a depth image such as an RGB-D sensor” (Yoshikawa, Page 16), a polarized image, and an estimated self-position of the moving body “the position / orientation estimation unit 103 stores the position / orientation information acquired in S302 in the camera attitude storage unit 112 (S303)” (Yoshikawa, Page 9).
Yoshikawa fails to explicitly teach of a polarized image.
Atsushi teaches a polarized image “The extraction unit 32 extracts the specular reflection area (generates a specular reflection area image) using a 45-degree polarized image with fewer specular reflection components than the one used to detect the specular reflection area” (Atsushi, Page 9) where a polarized image is used for accurate color and lighting.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of defect detection in an image taught by Yoshikawa with the use of a polarized image taught by Atsushi to include accurate lighting and color in the captured image. The suggestion/motivation to do so would have been to more accurately detect the image which is intended to be scanned.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Hideki Ohnishi Et. Al. (Pat. Pub. US-20110063456-A1) discloses an image capture apparatus and method designed to reduce defects in the captured image and produce more a more desired picture (abstract).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAIDEN ALEXANDER USSERY whose telephone number is (571)272-1192. The examiner can normally be reached Monday - Friday* 7:30AM - 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571) 272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.A.U./Examiner, Art Unit 2611
/TAMMY GODDARD/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights