DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/26/2025 has been entered.
Response to Arguments
Applicant's arguments filed 10/31/2025 have been fully considered.
Regarding to claim 1 and claim 19, the applicant argues that generating imaging assisting information for the object concerning imaging by an imaging unit for capturing the image, based on imaging range information corresponding to an updated provisional three-dimensional model, when the provisional three-dimensional model is updated. The arguments have been fully considered. The argument according “corresponding to an updated provisional three-dimensional model, when the provisional three-dimensional model is updated” is persuasive. Therefore, the 35 U.S.C 103 rejection has been withdrawn. However, upon further consideration, new grounds of rejection are made in newly applied art. The argument according “generating imaging assisting information for the object concerning imaging by an imaging unit for capturing the image, based on imaging range information” is not persuasive. The examiner cannot concur with the applicant for following reasons:
Okada discloses “generating imaging assisting information for the object concerning imaging by an imaging unit for capturing the image, based on imaging range information”. For example, in Fig. 21A-B and paragraph [0042], Okada teaches projecting, onto a floor surface, a guide image, e.g., an arrow, for guiding a user to a range where the sensing device 12 performs sensing; Okada further teaches projecting a CG image obtained by rendering a user's 3D model onto a wall surface, a screen, or the like;
PNG
media_image1.png
336
520
media_image1.png
Greyscale
. In paragraph [0046], Okada teaches displaying a guide and assisting image on the head-mounted display worn by a user. In Fig. 21 and paragraph [0072], Okada teaches controlling the projector 14 or the head-mounted display 16 to project a guide image, e.g., an arrow in FIG. 21, for guiding and assisting the user. In Fig. 21A-B and paragraph [0127], Okada teaches an arrow illustrated in FIG. 21 is a guide image projected onto a floor surface by the projector 14 to guide a user to a range where the sensing device 12 performs sensing and capturing images; Okada further teaches an arrow is presented to guide the user to the optimum position and orientation for capturing of a color image and a depth image by the sensing device 12. In paragraph [0128], Okada teaches reproducing the motion at the optimum position and orientation for sensing and capturing image by the sensing device 12.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7, and 10-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kurz (US 11935263 B1) in view of Okada (US 20210295538 A1), in view of Dougherty (US 20200312021 A1), and further in view of Yamamoto (US 20130136341 A1).
Regarding to claim 1 (Currently Amended), Kurz discloses an information processing apparatus configured to provide imaging assisting information for an object to a user of an image pickup apparatus to collect images for generating a three-dimensional model of the object, the information processing apparatus comprising (col. 5, lines 15-25: one or more cameras or other sensors capture data used to generate a three-dimensional (3D) model of the door 150, the window 160, and walls; col. 5, lines 63-67: image 200 is an image taken by user 102 using an image sensor of device 120; Fig. 3; col. 6, lines 35-50: determine directions 302, 304 for an image that is captured; the image 200 of FIG. 2 is used in generating a 3D model of one or more objects of the physical environment 100; determine direction 302 from the electronic device 120 to celestial element 170 at the time of image capture; Fig. 5; col. 9, lines 5-11: the view 500 includes a virtual compass 502 that may be displayed in the view 500 based on the determined global reference direction 515 of the physical environment 100; prove direction assisting information to user;
PNG
media_image2.png
118
172
media_image2.png
Greyscale
;
PNG
media_image3.png
134
186
media_image3.png
Greyscale
; Fig. 6; col. 9, lines 30-40: the view 600 includes a compass 602 that is displayed to show a global reference direction;
PNG
media_image4.png
388
530
media_image4.png
Greyscale
; provide direction assisting information):
a memory storing instructions (Fig. 8; col. 16, lines 25-40: the memory 820 includes high-speed random-access memory, such as DRAM, SRAM, and DDR RAM); and
a processor configured to execute the instructions to (Fig. 8; col. 16, lines 55-65: the global reference instruction set is executable by the processing units to implement a true north detection):
(1) acquire object information including an image of the object (Fig. 5; col. 5, lines 15-25: one or more cameras capture and acquire data used to generate a three-dimensional (3D) model of the door 150, the window 160, and walls; Fig. 2; col. 5, lines 60-67: the image 200 is an image taken by user 102 using an image sensor, e.g., RGB camera, of device 120; image 200 includes image content),
(2) acquire a provisional three-dimensional model (Fig. 7; col. 12, lines 50-60: obtains a 3D model of the physical environment; a 3D model may include a 3D reconstruction , e.g., a physical room, or an object, based on a textured mesh, a sparse point cloud, a SLAM/VIO map; col. 13, lines 40-50: merging multiple acquired local 3D models into a single global 3D model; obtain a second 3D model of at least part of the physical environment),
(3) calculate imaging range information including position-posture of the image pickup apparatus with respect to the object for the acquired image of the object and including a view angle range with respect to the object for the image of the object (col. 8, lines 15-25: the angle between two directions include the angle between the rays pointing to the two directions projected onto the ground plane; col. 10, lines 10-25: determine a camera's orientation relative to the direction of the sun based on where the sun is depicted in an image; determine the camera's orientation relative to true north; using this camera orientation relative to true north to orient 3D models generated using the camera's orientation relative to true north; Fig. 7; col. 12, lines 60-67: determine an orientation of the image sensor with respect to the 3D model; col. 13, lines 1-10: determine a current camera pose within the 3D model as part of a SLAM process or other 3D modeling process; col. 17, lines 6-20: track a location of a device, e.g. device 120, in a 3D coordinate system; track and calculate device location information for 3D model; col. 17, lines 25-35: obtain localization data from the from position tracking instruction set 844; obtain other sources of physical environment information, e.g., camera positioning information).
Kurz fails to explicitly disclose:
including an uncaptured part of the object upon inputting of the object information,
(4) generate the imaging assisting information for the object concerning imaging by an imaging unit for capturing the image, based on the imaging range information corresponding to an updated provisional three-dimensional model, when the provisional three-dimensional model is updated, and
(5) notify the imaging assisting information to the user,
wherein the imaging assisting information includes completion and incompletion rates of imaging for collecting the images for generating the three-dimensional model of the object.
In same field of endeavor, Okada teaches:
three-dimensional model including an uncaptured part of the object upon inputting of the object information (Fig. 3; [0051]: occlusion areas are not noticeable in the CG image illustrated in FIG. 3; occlusion areas are uncaptured part of the object; [0063]: a mesh represents a three-dimensional shape of the user; Fig. 18; [0123]: a texture acquisition state visualization map is created; a texture acquisition state visualization map includes each of an occlusion area, i.e. not yet images;
PNG
media_image5.png
108
446
media_image5.png
Greyscale
),
(4) generate the imaging assisting information for the object concerning imaging by an imaging unit for capturing the image, based on the imaging range information (Fig. 21A-B; [0042]: project, onto a floor surface, a guide image, e.g., an arrow, for guiding a user to a range where the sensing device 12 performs sensing; project a CG image obtained by rendering a user's 3D model onto a wall surface, a screen, or the like;
PNG
media_image1.png
336
520
media_image1.png
Greyscale
; [0046]: display a guide and assisting image on the head-mounted display worn by a user; Fig. 21; [0072]: controls the projector 14 or the head-mounted display 16 to project a guide image, e.g., an arrow in FIG. 21, for guiding and assisting the user; Fig. 21A-B; [0127]: an arrow illustrated in FIG. 21 is a guide image projected onto a floor surface by the projector 14 to guide a user to a range where the sensing device 12 performs sensing and capturing image; an arrow is presented to guide the user to the optimum position and orientation for capturing of a color image and a depth image by the sensing device 12; [0128]: reproduce the motion at the optimum position and orientation for sensing and capturing image by the sensing device 12.);
(5) notify the imaging assisting information to the user (Fig. 21A-B; [0042]: project, onto a floor surface, a guide image, e.g., an arrow, for guiding a user to a range where the sensing device 12 performs sensing; project a CG image obtained by rendering a user's 3D model onto a wall surface, a screen, or the like;
PNG
media_image1.png
336
520
media_image1.png
Greyscale
; [0046]: display a guide and assisting image on the head-mounted display worn by a user),
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kurz to include three-dimensional model including an uncaptured part of the object upon inputting of the object information, (4) generate the imaging assisting information for the object concerning imaging by an imaging unit for capturing the image, based on the imaging range information; (5) notify the imaging assisting information to the user as taught by Okada. The motivation for doing so would have been to display and project a guide image, e.g., an arrow, for guiding a user to a range where the sensing device 12 performs sensing; to display a guide image on the head-mounted display 16 worn by a user; to display an arrow to guide the user to the optimum position and orientation for capturing of a color image and a depth image by the sensing device as taught by Okada in Fig. 21 and paragraphs [0042], [0046], and [0127].
Kurz in view of Okada fails to explicitly disclose:
corresponding to an updated provisional three-dimensional model, when the provisional three-dimensional model is updated,
wherein the imaging assisting information includes completion and incompletion rates of imaging for collecting the images for generating the three-dimensional model of the object.
In same field of endeavor, Dougherty teaches:
wherein the imaging assisting information includes completion and incompletion rates of imaging for collecting the images for generating the three-dimensional model of the object ([0013]: display a three-dimensional (3D) reconstruction of the image data from the camera; request the user take additional photographs of a missing amenity; request the user take additional photographs for a specified area of the home; Fig. 7; [0058]: indicate one or more areas, e.g., 704, of the space for which image data has been captured, and one or more areas , e.g., 702, for which image data has not yet been captured;
PNG
media_image6.png
402
540
media_image6.png
Greyscale
; Fig. 12; [0080]: take a requested photograph and display guidance; rate is 3 out of 8 items;
PNG
media_image7.png
138
442
media_image7.png
Greyscale
).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kurz and Okada to include wherein the imaging assisting information includes completion and incompletion rates of imaging for collecting the images for generating the three-dimensional model of the object as taught by Dougherty. The motivation for doing so would have been to display a three-dimensional (3D) reconstruction of the image data from the camera; to request the user take additional photographs of a missing amenity; to request the user take additional photographs for a specified area of the home; to indicate one or more areas of the space for which image data has been captured, and one or more areas for which image data has not yet been captured; to lower the chances that an object is missed or skipped; to request that the user take a photograph of one or more missing amenities as taught by Dougherty in Fig, 7, Fig. 12, and paragraphs [0013], [0058], [0066], and [0079].
Kurz in view of Okada and Dougherty fails to explicitly disclose:
corresponding to an updated provisional three-dimensional model, when the provisional three-dimensional model is updated.
In same field of endeavor, Yamamoto teaches:
corresponding to an updated provisional three-dimensional model, when the provisional three-dimensional model is updated ([0042]: the 3D model data generator 34 updates the provisional 3D model data 109C stored in the data storage 109; the 3D model data generator 34 notifies the notification controller 35 that the provisional 3D model data 109C has been updated; [0043]: the notification controller 35 notifies the user of the position at which the object 2 is to be next captured, based on the provisional 3D model data 109C which has been updated by the 3D model data generator 34; the notification controller 35 notifies the user of the position at which the object 2 is to be next captured by information displayed on the screen of the LCD 17A; [0075]: the notification controller 35 detects a missing area of the provisional 3D model 2A by using the generated provisional 3D model data 109C).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kurz in view of Okada and Dougherty to include corresponding to an updated provisional three-dimensional model, when the provisional three-dimensional model is updated as taught by Yamamoto. The motivation for doing so would have been to estimate the position and posture of the camera 12 by using the coordinates on the image of the feature points; to notify the 3D model data generator 34 that the position and posture of the camera 12 have been estimated; to update the provisional 3D model data 109C stored in the data storage 109; to notify the user of the position at which the object 2 is to be next captured, based on the provisional 3D model data 109C which has been updated by the 3D model data generator 34; to detect a missing area of the provisional 3D model 2A by using the generated provisional 3D model data 109C as taught by Yamamoto in paragraphs [0040], [0042-0043], and [0075].
Regarding to claim 2 (Original), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the object information includes at least one of an image including at least part of the object (Kurz; Fig. 2; col. 5, lines 60-67: the image 200 is an image taken by user 102 using an image sensor, e.g., RGB camera, of device 120; image 200 includes image content; Fig. 7, col. 7, lines 35-50: obtain an image from the image sensor, wherein the image includes a depiction of a physical environment; the image 200 of FIG. 2 of the physical environment 100 includes a depiction 270 of the celestial element 170), GPS information and orientation information about the image pickup apparatus, voice information from the user, or text information from the user (or is optional; Kurz; col. 1, lines 60-67: longitude and latitude 65 of the camera, e.g., from GPS, Wi-Fi, etc., processes described herein can compute angles, e.g., azimuth; col. 10, lines 10-25: determine a camera's orientation relative to the direction of the sun based on where the sun is depicted in an image; determine the camera's orientation relative to true north; using this camera orientation relative to true north to orient 3D models generated using the camera's orientation relative to true north; Fig. 7; col. 12, lines 60-67: determine an orientation of the image sensor with respect to the 3D model; col. 13, lines 1-10: determine a current camera pose within the 3D model as part of a SLAM process or other 3D modeling process; col. 17, lines 6-20: track a location of a device, e.g. device 120, in a 3D coordinate system; track and calculate device location information for 3D model; col. 17, lines 25-35: the 3D model instruction set 846 obtains localization data from position tracking instruction set 844; col. 18, lines 20-35: headphones/earphones, speaker arrays; HMD and smartphone capture images or video of the physical environment; one or more microphones capture audio of the physical environment),
Kurz in view of Okada further discloses voice information from the user, or text information from the user (or is optional; Okada; [0140]: the input unit 107 is constituted by a keyboard, a mouse, a microphone).
The motivation of claim 1 is applied here.
Regarding to claim 3 (Original), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the processor is configured to generate the provisional three-dimensional model at a server or cloud outside the image pickup apparatus upon inputting of the object information (Kurz; col. 5, lines 45-55: the device 120 communicates with a separate controller or server to manage and coordinate an experience for the user; a controller or server are located in or may be remote relative to the physical environment 100; col. 10, lines 1-10: server device; col. 12, lines 60-67: the 3D model is generated and is provided by another type of device; another type of device includes a server).
Regarding to claim 4 (Original), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the object information is an image including at least part of the object (Kurz; Fig. 5; col. 5, lines 15-25: one or more cameras capture and acquire data used to generate a three-dimensional (3D) model of the door 150, the window 160, and walls; Fig. 2; col. 5, lines 60-67: the image 200 is an image taken by user 102 using an image sensor, e.g., RGB camera, of device 120; image 200 includes image content; Fig. 7, col. 7, lines 35-50: obtain an image from the image sensor, wherein the image includes a depiction of a physical environment; the image 200 of FIG. 2 of the physical environment 100 includes a depiction 270 of the celestial element 170), and
wherein the processor is configured to generate the provisional three-dimensional model of the entire object from the image including at least part of the object (Kurz; Fig. 5; col. 5, lines 15-25: one or more cameras capture and acquire data used to generate a three-dimensional (3D) model of the door 150, the window 160, and walls; col. 12, lines 50-60: a 3D model may include a 3D reconstruction, e.g., a physical room, or an object, based on a textured mesh, a sparse point cloud, a SLAM/VIO map, etc.; the 3D model of the physical environment is based on one or more images from the image sensor).
Regarding to claim 7 (Original), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the object information is at least one of text information or voice information (or is optional; Kurz; col. 17, lines 25-35: obtain other sources of physical environment information, and generate a 3D representation, e.g., a 3D mesh representation, a 3D point cloud with associated semantic labels, or the like, for an XR experience; semantic labels are text information; col. 18, lines 10-20: the XR system may adjust characteristic of graphical content in the XR environment in response to representations of physical motions, e.g., vocal commands; col. 18, lines 20-35: headphones/earphones, speaker arrays; HMD and smartphone capture images or video of the physical environment; one or more microphones capture audio of the physical environment; ), and
wherein the processor is configured to provide past provisional three-dimensional models accumulated in advance in association with at least one of the text information or the voice information (Kurz; col. 17, lines 25-35: obtains other sources of physical environment information, and generate a 3D representation, e.g., a 3D mesh representation, a 3D point cloud with associated semantic labels, or the like, for an XR experience; semantic labels are text information; col. 18, lines 10-20: the XR system may adjust characteristic of graphical content in the XR environment in response to representations of physical motions, e.g., vocal commands; col. 18, lines 20-35: headphones/earphones, speaker arrays; HMD and smartphone capture images or video of the physical environment; one or more microphones capture audio of the physical environment; col. 18, lines 45-50: project virtual objects into the physical environment as a hologram or on a physical surface).
Kurz in view of Okada, Dougherty, and Yamamoto further discloses wherein the object information is at least one of text information or voice information (Okada; Fig. 3; [0050]: the front side, the back side, and the right side; Fig. 7; [0096]: generate a texture from a right-side image and a front image of a user).
Same motivation of claim 1 is applied here.
Regarding to claim 10 (Original), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the processor calculates the imaging range information at a server or cloud outside the image pickup apparatus (or is optional; Kurz; col. 5, lines 45-55: the device communicates with a separate controller or server to manage and coordinate an experience for the user; such a controller or server are located in or may be remote relative to the physical environment 100; col. 10, lines 1-10: server device; col. 12, lines 60-67: the 3D model is generated and is provided by another type of device; another type of device includes a server).
Regarding to claim 11 (Original), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the processor acquires an image captured by the image pickup apparatus and transmits the image to another image pickup apparatus (Kurz; col. 5, lines 20-35: the electronic device 120 is moved about within the physical environment; other sensor data may be obtained and used to generate a point cloud, mesh, or other 3D model of one or more of the objects present in the physical environment; col. 5, lines 55-65: the multiple devices that may be used to accomplish the functions of electronic device 120 communicate with one another via wired or wireless communications; col. 12, lines 60-67: the 3D model is generated and provided by another type of device, e.g., a laser scanner, or by manually modeling the 3D model using 3D modeling software).
Regarding to claim 12 (Original), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the processor acquires an image captured by the image pickup apparatus and calculates the imaging range information at a server or cloud outside the image pickup apparatus (Kurz; col. 5, lines 45-55: the device communicates with a separate controller or server to manage and coordinate an experience for the user; such a controller or server may be located in or may be remote relative to the physical environment 100; col. 10, lines 1-10: server device; col. 12, lines 60-67: the 3D model is generated and is provided by another type of device; another type of device includes a server).
Regarding to claim 13 (Previously Presented), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the imaging assisting information further includes at least one of the position-posture or the view angle range (or is optional; Okada; Fig. 21A-B; [0042]: projects, onto a floor surface, a guide image for guiding a user to a range where the sensing device 12 performs sensing; project a CG image obtained by rendering a user's 3D model onto a wall surface, a screen, or the like; Fig. 21; [0072]: controls the projector 14 or the head-mounted display 16 to project a guide image for guiding the user; Fig. 21A-B; [0127]: an arrow is a guide image projected onto a floor surface by the projector 14 to guide a user to a range where the sensing device 12 can perform sensing).
Same motivation of claim 1 is applied here.
Regarding to claim 14 (Previously Presented), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the imaging assisting information further includes an instruction for reimaging at a closer distance or an instruction for reimaging with zoom-in in a case where imaging distance of the image acquired for the object by the processor to the object is longer than a predetermined distance (or is optional; Okada; Fig. 21A-B; [0042]: projects, onto a floor surface, a guide image for guiding a user to a range where the sensing device 12 can perform sensing; project a CG image obtained by rendering a user's 3D model onto a wall surface, a screen, or the like; move closer to camera as illustrated in Fig. 21A;
PNG
media_image8.png
342
534
media_image8.png
Greyscale
; Fig. 21 A; [0072]: controls the projector 14 or the head-mounted display 16 to project a guide image for guiding the user; Fig. 21A-B; [0127]: an arrow illustrated in FIG. 21 is a guide image projected onto a floor surface by the projector 14 to guide a user to a range where the sensing device 12 can perform sensing).
Same motivation of claim 1 is applied here.
Regarding to claim 15 (Previously Presented), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein in a case where a motion blur or a focal point blur occurs to the image acquired for the object by the processor (Okada; [0103]: a blur is left due to a high speed movement of a user; [0104]: there is a high possibility that a blur is left in textures of the sole of the foot, and it is determined that the textures have not been appropriately acquired; [0107]: occurrence of a blur is suppressed and higher quality textures are acquired), the imaging assisting information further includes an instruction for reimaging in position-posture close to the position-posture of the image pickup apparatus when the image of the object is captured (Okada; Fig. 21A-B; [0042]: projects, onto a floor surface, a guide image for guiding a user to a range where the sensing device 12 can perform sensing; project a CG image obtained by rendering a user's 3D model onto a wall surface, a screen, or the like; move closer to camera as illustrated in Fig. 21A;
PNG
media_image8.png
342
534
media_image8.png
Greyscale
; Fig. 21 A; [0072]: control the projector 14 or the head-mounted display 16 to project a guide image for guiding the user; Fig. 21A-B; [0127]: an arrow illustrated in FIG. 21 is a guide image projected onto a floor surface by the projector 14 to guide a user to a range where the sensing device 12 performs sensing).
Same motivation of claim 1 is applied here.
Regarding to claim 16 (Previously Presented), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the imaging assisting information further includes at least one of a notification that an accurate shape of the object cannot be acquired or an instruction for reimaging with a different imaging distance or imaging angle (Okada; Fig. 3; [0051]: occlusion areas, which are not noticeable in the CG image illustrated in FIG. 3; [0063]: a mesh represents a three-dimensional shape of the user; Fig. 18; [0123]: a texture acquisition state visualization map is created; a texture acquisition state visualization map includes each of an occlusion area, i.e. not yet images;
PNG
media_image5.png
108
446
media_image5.png
Greyscale
), the notification and the instruction being issued in a case where the shape of the provisional three-dimensional model generated for the object upon inputting of the image acquired for the object by the processor is deviated from the accurate shape (Okada; Fig. 3; [0051]: occlusion areas, which are not noticeable in the CG image illustrated in FIG. 3; [0063]: a mesh represents a three-dimensional shape of the user; Fig. 18; [0123]: a texture acquisition state visualization map is created; a texture acquisition state visualization map includes each of an occlusion area, i.e. not yet images;
PNG
media_image5.png
108
446
media_image5.png
Greyscale
).
Same motivation of claim 1 is applied here.
Regarding to claim 17 (Previously Presented), Kurz discloses an image pickup apparatus (col. 5, lines 15-25: one or more cameras or other sensors capture data used to generate a three-dimensional (3D) model of the door 150, the window 160, and walls; col. 5, lines 63-67: image 200 is an image taken by user 102 using an image sensor of device 120; Fig. 3; col. 6, lines 35-50: determine directions 302, 304 for an image that is captured; the image 200 of FIG. 2 is used in generating a 3D model of one or more objects of the physical environment 100; determine direction 302 from the electronic device 120 to celestial element 170 at the time of image capture; Fig. 5; col. 9, lines 5-11: the view 500 includes a virtual compass 502 that may be displayed in the view 500 based on the determined global reference direction 515 of the physical environment 100; prove direction assisting information to user;
PNG
media_image2.png
118
172
media_image2.png
Greyscale
;
PNG
media_image3.png
134
186
media_image3.png
Greyscale
; Fig. 6; col. 9, lines 30-40: the view 600 includes a compass 602 that is displayed to show a global reference direction;
PNG
media_image4.png
388
530
media_image4.png
Greyscale
; provide direction assisting information) comprising:
an imaging unit configured to capture an image of an object (Fig. 5; col. 5, lines 15-25: one or more cameras capture and acquire data used to generate a three-dimensional (3D) model of the door 150, the window 160, and walls; Fig. 7; col. 12, lines 50-60: obtains a 3D model of the physical environment; a 3D model may include a 3D reconstruction , e.g., a physical room, or an object, based on a textured mesh, a sparse point cloud, a SLAM/VIO map); and
wherein the imaging unit comprises a camera (Fig. 5; col. 5, lines 15-25: the electronic device 120 includes one or more cameras that are used to capture data).
the rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 17.
Regarding to claim 18 (Original), Kurz in view of Okada, Dougherty, and Yamamoto discloses the image pickup apparatus according to claim 17, further comprising an assisting information provision means for providing the imaging assisting information to the user (Okada; Fig. 21A-B; [0042]: projects, onto a floor surface, a guide image, e.g., an arrow, for guiding a user to a range where the sensing device 12 performs sensing; project a CG image obtained by rendering a user's 3D model onto a wall surface, a screen, or the like; [0046]: display a CG image, a guide image, on the head-mounted display worn by a user; Fig. 21; [0072]: control the projector 14 or the head-mounted display 16 to project a guide image, e.g., an arrow in FIG. 21 described later, for guiding the user; Fig. 21A-B; [0127]: an arrow illustrated in FIG. 21 is a guide image projected onto a floor surface by the projector 14 to guide a user to a range where the sensing device 12 can perform sensing).
Same motivation of claim 1 is applied here.
Regarding to claim 19 (Currently Amended), Kurz discloses an information processing method of providing imaging assisting information for an object to a user of an image pickup apparatus to collect images for generating a three-dimensional model of the object, the information processing method comprising steps, effected by a system comprising a processor and a memory, of (col. 5, lines 15-25: one or more cameras or other sensors capture data used to generate a three-dimensional (3D) model of the door 150, the window 160, and walls; col. 5, lines 63-67: image 200 is an image taken by user 102 using an image sensor of device 120; Fig. 3; col. 6, lines 35-50: determine directions 302, 304 for an image that is captured; the image 200 of FIG. 2 is used in generating a 3D model of one or more objects of the physical environment 100; determine direction 302 from the electronic device 120 to celestial element 170 at the time of image capture; Fig. 5; col. 9, lines 5-11: the view 500 includes a virtual compass 502 that may be displayed in the view 500 based on the determined global reference direction 515 of the physical environment 100; prove direction assisting information to user;
PNG
media_image2.png
118
172
media_image2.png
Greyscale
;
PNG
media_image3.png
134
186
media_image3.png
Greyscale
; Fig. 6; col. 9, lines 30-40: the view 600 includes a compass 602 that is displayed to show a global reference direction;
PNG
media_image4.png
388
530
media_image4.png
Greyscale
; provide direction assisting information; col. 10, lines 1-10: a processor executes one or more instruction sets stored in a non-transitory computer-readable medium, e.g., a memory; col. 10, lines 20-35):
The rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 19.
Regarding to claim 20 (Original), Kurz discloses a non-transitory computer-readable storage medium storing a computer program for causing a computer to execute the information processing method (col. 5, lines 15-25: one or more cameras or other sensors capture data used to generate a three-dimensional (3D) model of the door 150, the window 160, and walls; col. 5, lines 63-67: image 200 is an image taken by user 102 using an image sensor of device 120; Fig. 3; col. 6, lines 35-50: determine directions 302, 304 for an image that is captured; the image 200 of FIG. 2 is used in generating a 3D model of one or more objects of the physical environment 100; determine direction 302 from the electronic device 120 to celestial element 170 at the time of image capture; Fig. 5; col. 9, lines 5-11: the view 500 includes a virtual compass 502 that may be displayed in the view 500 based on the determined global reference direction 515 of the physical environment 100; prove direction assisting information to user;
PNG
media_image2.png
118
172
media_image2.png
Greyscale
;
PNG
media_image3.png
134
186
media_image3.png
Greyscale
; Fig. 6; col. 9, lines 30-40: the view 600 includes a compass 602 that is displayed to show a global reference direction;
PNG
media_image4.png
388
530
media_image4.png
Greyscale
; provide direction assisting information; Fig. 8; col. 16, lines 25-40: The memory 820 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM; Fig. 8; col. 16, lines 55-65: the global reference instruction set 842 is executable by the processing unit(s) 802 to implement a true north detection).
The rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 20.
Claims 5-6 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Kurz (US 11935263 B1) in view of Okada (US 20210295538 A1), in view of Dougherty (US 20200312021 A1), in view of Yamamoto (US 20130136341 A1), and further in view of Thomas (US 20210142577 A1).
Regarding to claim 5 (Original), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 4,
Kurz in view of Okada, Dougherty, and Yamamoto fails to explicitly disclose wherein the processor is configured to generate the provisional three-dimensional model of the entire object from the image including at least part of the object by using deep learning.
In same field of endeavor, Thomas teaches wherein the processor is configured to generate the provisional three-dimensional model of the entire object from the image including at least part of the object by using deep learning (Fig. 11; [0087]: 3D surfaces in a virtual space; a surface orientation color map 1100 is represented by 3D surfaces in a virtual space; a 2D image associated with the 3D geometric data representing surface orientation color map 1100 is inputted into one or more trained machine-learning models, such as a deep neural network including a pipeline of convolutional neural networks, to generate an output as in FIG. 11 expressing a surface normal prediction for each surface).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kurz in view of Okada, Dougherty, and Yamamoto to include wherein the processor is configured to generate the provisional three-dimensional model of the entire object from the image including at least part of the object by using deep learning as taught by Thomas. The motivation for doing so would have been to improve the performance of image processing using computing resources; to project the synthetic image data onto the depicted physical structure; to input a 2D image associated with the 3D geometric data representing surface orientation color map 1100 into one or more trained machine-learning models, such as a deep neural network including a pipeline of convolutional neural networks, to generate an output as in FIG. 11 expressing a 3D surface normal prediction for each surface as taught by Thomas in paragraphs [0027-0028], and [0087].
Regarding to claim 6 (Original), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the object information is GPS information and orientation information about the image pickup apparatus (Kurz; col. 1, lines 60-67: longitude and latitude 65 of the camera, e.g., from GPS, Wi-Fi, etc., processes described herein can compute angles, e.g., azimuth; Fig. 7; col. 12, lines 60-67: determine an orientation of the image sensor with respect to the 3D model; col. 13, lines 1-10: determine a current camera pose within the 3D model as part of a SLAM process or other 3D modeling process), and
Kurz in view of Okada, Dougherty, and Yamamoto fails to explicitly disclose:
wherein the processor is configured to provide past provisional three-dimensional models accumulated in advance in association with the GPS information and orientation information.
In same field of endeavor, Thomas teaches:
wherein the processor is configured to provide past provisional three-dimensional models accumulated in advance in association with the GPS information and orientation information ([0058]: select portions of the synthetic 3D model; the ambient light effect, such as stored in the report for that camera position, may be similarly applied to the synthetic 3D model selection to impart the same conditions as in the original input image; for a given GPS location at a given time of day, sunlight information such as direction and brightness are derived and applied to the synthetic 3D geometry; [0076]: effects incident to the image input are applied to the synthetic 3D model geometry; [0077]: the synthetic 3D model features are selectively displayed with the video of the image source input).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kurz in view of Okada, Dougherty, and Yamamoto to include wherein the processor is configured to provide past provisional three-dimensional models accumulated in advance in association with the GPS information and orientation information as taught by Thomas. The motivation for doing so would have been to improve the performance of image processing using computing resources; to project the synthetic image data onto the depicted physical structure; to input a 2D image associated with the 3D geometric data representing surface orientation color map 1100 into one or more trained machine-learning models, such as a deep neural network including a pipeline of convolutional neural networks, to generate an output as in FIG. 11 expressing a 3D surface normal prediction for each surface as taught by Thomas in paragraphs [0027-0028], and [0087].
Regarding to claim 9 (Original), Kurz in view of Okada, Dougherty, Yamamoto and Thomas discloses the information processing apparatus according to claim 6, wherein the past provisional three-dimensional models accumulated in advance includes at least one of a three-dimensional model generated by collecting data acquired from the image pickup apparatus (Or is optional; Kurz; Fig. 5; col. 5, lines 15-25: one or more cameras capture and acquire data used to generate a three-dimensional (3D) model of the door 150, the window 160, and walls; Fig. 2; col. 5, lines 60-67: the image 200 is an image taken by user 102 using an image sensor, e.g., RGB camera, of device 120; image 200 includes image content; col. 12, lines 60-67: the 3D model is generated and is provided by another type of device; a laser scanner) or a three-dimensional model generated by collecting data by using, instead or in addition, another image pickup apparatus or a depth sensor (Or is optional; Kurz; Fig. 5; col. 5, lines 15-25: one or more cameras capture and acquire data used to generate a three-dimensional (3D) model of the door 150, the window 160, and walls; Fig. 2; col. 5, lines 60-67: the image 200 is an image taken by user 102 using an image sensor, e.g., RGB camera, of device 120; image 200 includes image content; col. 10, lines 10-25: using this camera orientation relative to true north to orient 3D models generated using the camera's orientation relative to true north; col. 12, lines 60-67: the 3D model is generated and is provided by another type of device; by manually modeling the 3D model using 3D modeling software).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Kurz (US 11935263 B1) in view of Okada (US 20210295538 A1), in view of Dougherty (US 20200312021 A1), in view of Yamamoto (US 20130136341 A1), and further in view of Tsubaki (US 20140267809 A1).
Regarding to claim 8 (Original), Kurz in view of Okada, Dougherty, and Yamamoto discloses the information processing apparatus according to claim 1, wherein the object information is an image including at least part of the object (Kurz; Fig. 2; col. 5, lines 60-67: the image 200 is an image taken by user 102 using an image sensor, e.g., RGB camera, of device 120; image 200 includes image content; Fig. 7, col. 7, lines 35-50: obtain an image from the image sensor, wherein the image includes a depiction of a physical environment; the image 200 of FIG. 2 of the physical environment 100 includes a depiction 270 of the celestial element 170), and
Kurz in view of Okada, Dougherty, and Yamamoto fails to explicitly disclose:
wherein the processor is configured to acquire the provisional three-dimensional model by searching, with the image including at least part of the object as key information, past provisional three-dimensional models accumulated in advance in association with the image including at least part of the object.
In same field of endeavor, Tsubaki teaches:
wherein the processor is configured to acquire the provisional three-dimensional model by searching, with the image including at least part of the object as key information, past provisional three-dimensional models accumulated in advance in association with the image including at least part of the object (Fig. 4; [0039]: a first image 401 on the left side of FIG. 4 is a reference image and a second image 402 on the right side of FIG. 4 is an image to be searched; any search region 407 is set in the search image and processing for searching a position at which the template 403 matches most closely is executed while moving the search region 407 in sequence; [0042]: a motion vector is represented by a vector whose start point is the position of the point of interest 404 in each reference image and whose end point is the position of the corresponding point in the search image 402; [0045]: perform vector search using the position as a point of interest; [0046]: carry out a three-dimensional geometric transform with respect to the image capturing position of another image selected as a reference.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kurz in view of Okada, Dougherty, and Yamamoto to include wherein the processor is configured to acquire the provisional three-dimensional model by searching, with the image including at least part of the object as key information, past provisional three-dimensional models accumulated in advance in association with the image including at least part of the object as taught by Tsubaki. The motivation for doing so would have been to search a position at which the template 403 matches most closely is executed while moving the search region 407 in sequence; to perform vector search using the position as a point of interest; to carry out a three-dimensional geometric transform with respect to the image capturing position of another image selected as a reference; to improve the range finding performance as taught by Tsubaki in paragraphs [0039-0046] and [0090].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hai Tao Sun whose telephone number is (571)272-5630. The examiner can normally be reached 9:00AM-6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 5712727642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAI TAO SUN/Primary Examiner, Art Unit 2616
Prosecution Insights