Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-14 are rejected under 35 U.S.C. 103 as being unpatentable over IWAMI et al (EP 4109413) in view of Gamefromscratch (Meshroom -- 3D Models from Photos using this Free Open Source Photogrammetry Software – called MESHROOM), MA et al (US 20250390984), and PIIPPO et al (US 20120335446).
As per claim 1, Iwami teaches the claimed “information processing apparatus,” comprising circuitry configured to: “associate first position information indicating a first position in a captured image with second position information indicating a second position in a three-dimensional image, the captured image being obtained by capturing an image of a target object, the three- dimensional image including a three-dimensional area corresponding to the target object” (Iwami, [0032]-[0033] - Fig. 3 is a schematic diagram for explaining that the image data 5 and the point cloud data 7 have a corresponding positional relationship… Fig. 3 illustrates a pixel P having coordinates (Px, Py) on the image data 5A and a point Q having a corresponding positional relationship with the pixel P). It is noted that Iwari does not explicitly teach the simultaneously displays of the captured image and the associated 3D image as claimed. However, Iwari’s capture and generate the 2D image and the 3D image simultaneously (e.g., Iwami, [0031] - The three-dimensional measuring device 1 can acquire the point cloud data 7 and the image data 5 simultaneously or serially; [0045] - The display control unit 19 displays the point cloud data 7 stored in the storage unit 21 on the monitor 9) suggests a split-screen for displaying both the captured 2D image and its associated 3D image which shows the claimed “generate a screen including a first display area and a second display area, the first display area displaying a predetermined-area image that is a predetermined area in the captured image, the second display area displaying at least a part of the three-dimensional image in which the second position is associated with the first position in the captured image” (see also Piippo, [0032] - the 3D models might also be acquired separately and the range finder information used only to align the models with the panoramic images; Ma, [0058] - the picture finally displayed to the user is the to-be-displayed picture region, the target picture region, or both; Meshroom, 04:38 – the screen includes both the captured 2D image and its associated 3D image
PNG
media_image1.png
696
1223
media_image1.png
Greyscale
). Thus, it would have been obvious, in view of Piippo, Ma, and Meshroom, to configure Iwami’s apparatus as claimed by providing a split-screen to display simultaneously both the captured 2D image and its associated 3D image. The motivation is to improve the visual representation of a captured image by simultaneous showing its associated 3D object for enhancing the geographical and spatial displayed scene.
Claim 2 adds into claim 1 “wherein the circuitry is further configured to align a position of a second virtual camera with an image capturing position for the captured image in displaying the three-dimensional image in the second display area and align a first field of view for displaying the predetermined-area image in the first display area with a second field of view for displaying the three-dimensional image in the second display area to generate the three-dimensional image to be displayed in the second display area, the second field of view being a virtual field of view” (Ma, [0057] - In the context, the term "target picture region" may refer to a portion of the panoramic image from the first panoramic picture that contains the target object of interest, which is mapped to a target planar sub-window for focused display and recognition. In contrast, the term "to-be-displayed picture region" may refer to a general area of the panoramic image intended for display on the display interface, providing a broader scene context; [0058] - the picture finally displayed to the user is the to-be-displayed picture region, the target picture region, or both; Meshroom, 17:57 – camera’s Initialization includes viewpoint or extrinsic parameter selections; e.g., 50 captured images associated to 50 viewpoints of the virtual camera;
PNG
media_image2.png
691
1231
media_image2.png
Greyscale
). Thus, it would have been obvious, in view of Piippo, Ma, and Meshroom, to configure Iwami’s apparatus as claimed by providing a split-screen to display simultaneously both the captured 2D image and its associated 3D image based on the viewpoints of the virtual cameras. The motivation is to improve the visual representation of a captured image by simultaneous showing its associated 3D object for enhancing the geographical and spatial displayed scene.
Claim 8 adds into claim 2 “wherein the captured image is a wide-field image having a wide field of view, and the circuitry is further configured to: acquire a change in the first field of view for the wide-field image” (Ma, Abstract - A first panoramic picture is acquired, and a to-be-displayed picture region and a target picture region are determined in the first panoramic picture); and “change the second field of view according to the change in the first field of view to change the three-dimensional image displayed in the second display area” (Ma, [0057] - In the context, the term "target picture region" may refer to a portion of the panoramic image from the first panoramic picture that contains the target object of interest, which is mapped to a target planar sub-window for focused display and recognition. In contrast, the term "to-be-displayed picture region" may refer to a general area of the panoramic image intended for display on the display interface, providing a broader scene context; [0058] - the picture finally displayed to the user is the to-be-displayed picture region, the target picture region, or both). Thus, it would have been obvious, in view of Piippo, Ma, and Meshroom, to configure Iwami’s apparatus as claimed by providing a split-screen to display simultaneously both the captured 2D image and its associated 3D image based on the viewpoints of the virtual cameras. The motivation is to improve the visual representation of a captured image by simultaneous showing its associated 3D object for enhancing the geographical and spatial displayed scene.
Claim 10 adds into claim 2 “wherein the circuitry is further configured to: acquire an instruction to change the second field of view for the three-dimensional image, and change the three-dimensional image displayed in the second display area according to a change in the second field of view” (Iwami, [0032]-[0033] - Fig. 3 is a schematic diagram for explaining that the image data 5 and the point cloud data 7 have a corresponding positional relationship… Fig. 3 illustrates a pixel P having coordinates (Px, Py) on the image data 5A and a point Q having a corresponding positional relationship with the pixel P).
Claim 11 adds into claim 10 “wherein the circuitry is configured to change the first field of view according to the change in the second field of view to change the predetermined-area image displayed in the first display area” (Iwami, [0032]-[0033] - Fig. 3 is a schematic diagram for explaining that the image data 5 and the point cloud data 7 have a corresponding positional relationship… Fig. 3 illustrates a pixel P having coordinates (Px, Py) on the image data 5A and a point Q having a corresponding positional relationship with the pixel P).
Claim 12 adds into claim 11 “wherein the circuitry is configured to: enlarge the predetermined-area image displayed in the first display area when the change in the second field of view is an enlargement of the three-dimensional image displayed in the second display area; and reduce the predetermined-area image displayed in the first display area when the change in the second field of view is a reduction of the three-dimensional image displayed in the second display area” (Piippo, Figures 4E-4F, [0067]-[0068] - As shown in FIGS. 4E-4F, the extent of viewing of the panoramic image can be adjusted by the user by widening the view sector as presented in connection with user interface element 407. By way of example, in FIG. 4E, a first view sector (e.g., triangle 417) corresponding a first field-of-view may be adapted, for example, through two-finger interaction as depicted in FIG. 4F. Under this scenario, the user places their fingers along opposing edges of the user interface element 407 to and moves their fingers away ( outward). As a result, a second view section (e.g., triangle 419) is generated corresponding to a second field-of-view. Of note, widening of the view sector also widens the user interface element 403. This enables more of the panoramic image 421 to be revealed). Thus, it would have been obvious, in view of Piippo, Ma, and Meshroom, to configure Iwami’s apparatus as claimed by providing a split-screen to display simultaneously both the captured 2D image and its associated 3D image based on the enlargement or reduction of the captured 2D image. The motivation is to improve the visual representation of a captured image by simultaneous showing its associated 3D object for enhancing the geographical and spatial displayed scene.
Claim 3 adds into claim 1 “wherein the circuitry is configured to generate the screen to display, in the second display area, a corresponding predetermined area that corresponds to the predetermined area of the predetermined-area image displayed in the first display area” (Iwami, [0032]-[0033] - Fig. 3 is a schematic diagram for explaining that the image data 5 and the point cloud data 7 have a corresponding positional relationship… Fig. 3 illustrates a pixel P having coordinates (Px, Py) on the image data 5A and a point Q having a corresponding positional relationship with the pixel P; Piippo, [0032] - the 3D models might also be acquired separately and the range finder information used only to align the models with the panoramic images; Ma, [0057] - In the context, the term "target picture region" may refer to a portion of the panoramic image from the first panoramic picture that contains the target object of interest, which is mapped to a target planar sub-window for focused display and recognition. In contrast, the term "to-be-displayed picture region" may refer to a general area of the panoramic image intended for display on the display interface, providing a broader scene context; [0058] - the picture finally displayed to the user is the to-be-displayed picture region, the target picture region, or both; Meshroom, 04:38 – the screen includes both the captured 2D image and its associated 3D image). Thus, it would have been obvious, in view of Piippo, ma, and Meshroom, to configure Iwari’s apparatus as claimed by providing a split-screen to display simultaneously both the captured 2D image and its associated 3D image. The motivation is to improve the visual representation of a captured image by simultaneous showing its associated 3D object for enhancing the geographical and spatial displayed scene.
Claim 4 adds into claim 1 “wherein the circuitry is configured to generate the screen to display a position image in the second display area, the position image indicating one of the captured image and an image capturing position for the captured image and being at a corresponding one of a position corresponding to the captured image and the image capturing position” (Iwami, [0032]-[0033] - Fig. 3 is a schematic diagram for explaining that the image data 5 and the point cloud data 7 have a corresponding positional relationship… Fig. 3 illustrates a pixel P having coordinates (Px, Py) on the image data 5A and a point Q having a corresponding positional relationship with the pixel P; Piippo, [0032] - the 3D models might also be acquired separately and the range finder information used only to align the models with the panoramic images; Ma, [0057] - In the context, the term "target picture region" may refer to a portion of the panoramic image from the first panoramic picture that contains the target object of interest, which is mapped to a target planar sub-window for focused display and recognition. In contrast, the term "to-be-displayed picture region" may refer to a general area of the panoramic image intended for display on the display interface, providing a broader scene context; [0058] - the picture finally displayed to the user is the to-be-displayed picture region, the target picture region, or both; Meshroom, 04:38 – the screen includes both the captured 2D image and its associated 3D image). Thus, it would have been obvious, in view of Piippo, ma, and Meshroom, to configure Iwari’s apparatus as claimed by providing a split-screen to display simultaneously both the captured 2D image and its associated 3D image. The motivation is to improve the visual representation of a captured image by simultaneous showing its associated 3D object for enhancing the geographical and spatial displayed scene.
Claim 5 adds into claim 4 “wherein the screen includes the second display area and initially excludes the first display area, and when an operation on the position image is received, the circuitry is configured to generate the screen to include the first display area” which is obvious as a conventional arrangement of split-screen display in which a general area (i.e., the claimed second region) is displayed first, then a sub-region is decided and subsequently displayed (as the first display area).
Claim 6 adds into claim 4 “wherein the circuitry is configured to generate the screen to display, as the position image, an image indicating a first virtual camera having an image capturing area corresponding to a field of view for displaying the predetermined-area image in the first display area” (Meshroom, 17:57 – camera’s Initialization includes viewpoint or extrinsic parameter selections
PNG
media_image1.png
696
1223
media_image1.png
Greyscale
).
Claim 7 adds into claim 6 “wherein the circuitry is configured to generate the screen to display a line of sight from the image of the first virtual camera toward a center point in the second display area” which Iwami suggests in figures 1 and 2 (e.g., Iwari, [0031] - The three-dimensional measuring device 1 can acquire the point cloud data 7 and the image data 5 simultaneously or serially; [0045] - The display control unit 19 displays the point cloud data 7 stored in the storage unit 21 on the monitor 9) (Meshroom, Meshroom, 02:01 – e.g., 50 captured images associated to 50 viewpoints of the virtual camera
PNG
media_image3.png
692
1235
media_image3.png
Greyscale
). Thus, it would have been obvious, in view of Piippo, ma, and Meshroom, to configure Iwari’s apparatus as claimed by providing a split-screen to display simultaneously both the captured 2D image and its associated 3D image displayed based on a line of sight (indicated a selection) of the virtual camera. The motivation is to improve the visual representation of a captured image by simultaneous showing its associated 3D object for enhancing the geographical and spatial displayed scene.
Claim 9 adds into claim 1 “wherein the circuitry is further configured to: acquire an instruction to change a viewpoint for the three-dimensional image, and change the three-dimensional image displayed in the second display area according to a change in the viewpoint” (Iwari, [0031] - The three-dimensional measuring device 1 can acquire the point cloud data 7 and the image data 5 simultaneously or serially; [0045] - The display control unit 19 displays the point cloud data 7 stored in the storage unit 21 on the monitor 9; Ma, [0057] - In the context, the term "target picture region" may refer to a portion of the panoramic image from the first panoramic picture that contains the target object of interest, which is mapped to a target planar sub-window for focused display and recognition. In contrast, the term "to-be-displayed picture region" may refer to a general area of the panoramic image intended for display on the display interface, providing a broader scene context; [0058] - the picture finally displayed to the user is the to-be-displayed picture region, the target picture region, or both). Thus, it would have been obvious, in view of Piippo, ma, and Meshroom, to configure Iwari’s apparatus as claimed by providing a split-screen to display simultaneously both the captured 2D image and its associated 3D image displayed based on viewpoint of the virtual camera. The motivation is to improve the visual representation of a captured image by simultaneous showing its associated 3D object for enhancing the geographical and spatial displayed scene.
Claims 13 and 14 claim a screen generation method and a non-transitory recording medium storing a plurality of instructions based on the apparatus of claims 1-12; therefore, they are rejected under a similar rationale.
Claim 15 adds into claim 1 “a display terminal communicably connected to the information processing apparatus and including a display to display the screen” (Iwari, [0045] - The display control unit 19 displays the point cloud data 7 stored in the storage unit 21 on the monitor 9; Ma, [0058] - the picture finally displayed to the user is the to-be-displayed picture region, the target picture region, or both). Thus, it would have been obvious, in view of Piippo, ma, and Meshroom, to configure Iwari’s apparatus as claimed by providing a split-screen to display simultaneously both the captured 2D image and its associated 3D image. The motivation is to improve the visual representation of a captured image by simultaneous showing its associated 3D object for enhancing the geographical and spatial displayed scene.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHU K NGUYEN whose telephone number is (571)272-7645. The examiner can normally be reached M-F 8-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHU K NGUYEN/Primary Examiner, Art Unit 2616
Prosecution Insights