DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-11, and 13-21 are pending
Claim 12 is cancelled
Claim 2-11, and 15-21 are amended.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 13, and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Izumi et al. (US 20200359007) in view of Xiao et al. (WO 2015188684).
Regarding claim 1.
Izumi teaches:
A walkthrough view generation method (Izumi [Overview] [Problem to be Solved] To provide an information processing apparatus and an information processing method.),
determining a first intersection-point set between walkthrough light rays corresponding to current walkthrough parameters and the initial three-dimensional model and a second intersection-point set between the walkthrough light rays and the repaired three-dimensional model respectively, wherein the current walkthrough parameters comprise a walkthrough viewing position after moving and a walkthrough viewing angle after moving (Izumi [0098] Here, it looks to the user like the texture at the point P.sub.L and the texture at the point P.sub.R fuse at the position of the point P at which the above-described two light rays intersect, and the surface of the three-dimensional model M32 is present at the position of the point P. [0127] For example, the HMD 3 finds a convenient use when allowing a user to freely walk around within some range. [0090] The positional relationship between the stereo camera and the three-dimensional object corresponds to the positional relationship between the stereo camera and the three-dimensional model in the three-dimensional space. The known positional relationship between the stereo camera and the three-dimensional model in three-dimensional space makes it possible to generate the texture data corresponding to the three-dimensional model as follows from a camera image acquired by the stereo camera. [0099] Moreover, in a case of viewing and listening with the camera positions matching with the positions of the eyes of a user, the light rays at the time of imaging by the cameras are reproduced and the true shape looks reproduced to the user even if the three-dimensional model M32 has any shape. [0107] The data set generation unit 11 generates a plurality of data sets having different data amounts on the basis of the original three-dimensional shape data, the original texture data, the left camera image, and the right camera image as described with reference to FIG. 2. [Figure 14 and Figure 15 [0096] The respective arrows extending from the left camera C31L and the right camera C31R to the point P thus intersect at not the point P, but a point P.sub.R and a point P.sub.L on the surface of the three-dimensional model M32. Distance d between these point P.sub.R and point P.sub.L is mapped as it is as the positional difference between the left-eye texture data and the right-eye texture data, and recorded as it is as left-right parallax. [Figure 13. P and P’ intersection points] [0100] It is not, however, possible to reproduce the actual light rays if the shape of the three-dimensional model M32 includes an error in a case of viewing and listening at positions different from the camera positions. FIG. 13 is a schematic diagram illustrating a viewing and listening situation in which the camera position and the position of an eye of a user are different from each other. In the situation illustrated in FIG. 13, the texture at the point P.sub.L appearing in the left eye E32L of a user and the texture at the point P.sub.L appearing in the right eye E32R of the user fuse at a point P′, and the surface of the three-dimensional model M32 looks present at the position of the point P′. Here, the point P′ illustrated in FIG. 13 is a position different from the point P that is the true tip position of the protrusion. This is because the three-dimensional model M32 includes an error. The point P′, however, looks present at a protruding position as compared with a case where the protrusion looks absent. This suppresses a decrease in the subjective image quality caused by an error even in a case of viewing and listening at positions different from the camera positions.);
Izumi fails to teach:
comprising: acquiring an initial three-dimensional model and a repaired three-dimensional model corresponding to the initial three-dimensional model in a same spatial region, wherein the repaired three-dimensional model is obtained by repairing spatial information in the initial three-dimensional model (Xiao [Pg 1 Par 3] The use of computer technology to model real objects has important significance in various fields. For example, the reconstruction of the three-dimensional model of the human body can reconstruct the posture, motion and topography of the human body in the computer for subsequent Provides a basis for applications such as gesture recognition and replacement of topographical features. [Pg 3 Par 20] In the three-dimensional model reconstruction system of the present invention, the model reconstructor performs surface fusion and hole filling on all the data after registration.);
and fusing the initial three-dimensional model and the repaired three-dimensional model according to depth differences between intersection-points of the first intersection-point set and corresponding intersection-points of the second intersection-point set and rendering a fused result to obtain a current walkthrough view (Izumi [0082] As described above, the use of stereo texture data for rendering the texture of a common three-dimensional model allows the rendered stereo display image to fuse at a position different from the surface of the three-dimensional model.) (Xiao [Abstract] The present invention relates to a three-dimensional model reconstruction method and system, the method comprising the following steps: S1, conducting image acquisition on a target using at least one depth camera to obtain a depth image of the target;).
Xiao teaches:
comprising: acquiring an initial three-dimensional model and a repaired three-dimensional model corresponding to the initial three-dimensional model in a same spatial region, wherein the repaired three-dimensional model is obtained by repairing spatial information in the initial three-dimensional model (Xiao [Pg 1 Par 3] The use of computer technology to model real objects has important significance in various fields. For example, the reconstruction of the three-dimensional model of the human body can reconstruct the posture, motion and topography of the human body in the computer for subsequent Provides a basis for applications such as gesture recognition and replacement of topographical features. [Pg 3 Par 20] In the three-dimensional model reconstruction system of the present invention, the model reconstructor performs surface fusion and hole filling on all the data after registration.);
Xiao and Izumi teach:
and fusing the initial three-dimensional model and the repaired three-dimensional model according to depth differences between intersection-points of the first intersection-point set and corresponding intersection-points of the second intersection-point set and rendering a fused result to obtain a current walkthrough view (Izumi [0082] As described above, the use of stereo texture data for rendering the texture of a common three-dimensional model allows the rendered stereo display image to fuse at a position different from the surface of the three-dimensional model.) (Xiao [Abstract] The present invention relates to a three-dimensional model reconstruction method and system, the method comprising the following steps: S1, conducting image acquisition on a target using at least one depth camera to obtain a depth image of the target;).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi with Xiao. Obtaining a 3d model and a repaired model and combining, as in Xiao, would benefit the Izumi teachings by allowing a way to combine images or different types of models. Additionally, this is the application of a known technique, 3d fusing models, to yield predictable results.
Regarding claim 13.
Izumi teaches:
A walkthrough view generation device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, performs (Izumi [0163] As illustrated in FIG. 26, the information processing apparatus 900 includes CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, RAM (Random Access Memory) 903, and a host bus 904a. In addition, the information processing apparatus 900 includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a coupling port 911, a communication device 913, and a sensor 915. The information processing apparatus 900 may include a processing circuit such as DSP or ASIC in place of or in addition to the CPU 901.):
determining a first intersection-point set between walkthrough light rays corresponding to current walkthrough parameters and the initial three-dimensional model and a second intersection-point set between the walkthrough light rays and the repaired three-dimensional model respectively, wherein the current walkthrough parameters comprise a walkthrough viewing position after moving and a walkthrough viewing angle after moving (Izumi [0098] Here, it looks to the user like the texture at the point P.sub.L and the texture at the point P.sub.R fuse at the position of the point P at which the above-described two light rays intersect, and the surface of the three-dimensional model M32 is present at the position of the point P. [0127] For example, the HMD 3 finds a convenient use when allowing a user to freely walk around within some range. [0090] The positional relationship between the stereo camera and the three-dimensional object corresponds to the positional relationship between the stereo camera and the three-dimensional model in the three-dimensional space. The known positional relationship between the stereo camera and the three-dimensional model in three-dimensional space makes it possible to generate the texture data corresponding to the three-dimensional model as follows from a camera image acquired by the stereo camera. [0099] Moreover, in a case of viewing and listening with the camera positions matching with the positions of the eyes of a user, the light rays at the time of imaging by the cameras are reproduced and the true shape looks reproduced to the user even if the three-dimensional model M32 has any shape. [0107] The data set generation unit 11 generates a plurality of data sets having different data amounts on the basis of the original three-dimensional shape data, the original texture data, the left camera image, and the right camera image as described with reference to FIG. 2. [Figure 14 and Figure 15 [0096] The respective arrows extending from the left camera C31L and the right camera C31R to the point P thus intersect at not the point P, but a point P.sub.R and a point P.sub.L on the surface of the three-dimensional model M32. Distance d between these point P.sub.R and point P.sub.L is mapped as it is as the positional difference between the left-eye texture data and the right-eye texture data, and recorded as it is as left-right parallax. [Figure 13. P and P’ intersection points] [0100] It is not, however, possible to reproduce the actual light rays if the shape of the three-dimensional model M32 includes an error in a case of viewing and listening at positions different from the camera positions. FIG. 13 is a schematic diagram illustrating a viewing and listening situation in which the camera position and the position of an eye of a user are different from each other. In the situation illustrated in FIG. 13, the texture at the point P.sub.L appearing in the left eye E32L of a user and the texture at the point P.sub.L appearing in the right eye E32R of the user fuse at a point P′, and the surface of the three-dimensional model M32 looks present at the position of the point P′. Here, the point P′ illustrated in FIG. 13 is a position different from the point P that is the true tip position of the protrusion. This is because the three-dimensional model M32 includes an error. The point P′, however, looks present at a protruding position as compared with a case where the protrusion looks absent. This suppresses a decrease in the subjective image quality caused by an error even in a case of viewing and listening at positions different from the camera positions.); and
Izumi fails to teach:
acquiring an initial three-dimensional model and a repaired three-dimensional model corresponding to the initial three-dimensional model in a same spatial region, wherein the repaired three-dimensional model is obtained by repairing spatial information in the initial three-dimensional model (Xiao [Pg 1 Par 3] The use of computer technology to model real objects has important significance in various fields. For example, the reconstruction of the three-dimensional model of the human body can reconstruct the posture, motion and topography of the human body in the computer for subsequent Provides a basis for applications such as gesture recognition and replacement of topographical features. [Pg 3 Par 20] In the three-dimensional model reconstruction system of the present invention, the model reconstructor performs surface fusion and hole filling on all the data after registration.);
fusing the initial three-dimensional model and the repaired three-dimensional model according to depth differences between intersection-points of the first intersection-point set and corresponding intersection-points of the second intersection-point set and rendering a fused result to obtain a current walkthrough view (Izumi [0082] As described above, the use of stereo texture data for rendering the texture of a common three-dimensional model allows the rendered stereo display image to fuse at a position different from the surface of the three-dimensional model.) (Xiao [Abstract] The present invention relates to a three-dimensional model reconstruction method and system, the method comprising the following steps: S1, conducting image acquisition on a target using at least one depth camera to obtain a depth image of the target;).
Xiao teaches:
acquiring an initial three-dimensional model and a repaired three-dimensional model corresponding to the initial three-dimensional model in a same spatial region, wherein the repaired three-dimensional model is obtained by repairing spatial information in the initial three-dimensional model (Xiao [Pg 1 Par 3] The use of computer technology to model real objects has important significance in various fields. For example, the reconstruction of the three-dimensional model of the human body can reconstruct the posture, motion and topography of the human body in the computer for subsequent Provides a basis for applications such as gesture recognition and replacement of topographical features. [Pg 3 Par 20] In the three-dimensional model reconstruction system of the present invention, the model reconstructor performs surface fusion and hole filling on all the data after registration.);
Xiao and Izumi teach:
fusing the initial three-dimensional model and the repaired three-dimensional model according to depth differences between intersection-points of the first intersection-point set and corresponding intersection-points of the second intersection-point set and rendering a fused result to obtain a current walkthrough view (Izumi [0082] As described above, the use of stereo texture data for rendering the texture of a common three-dimensional model allows the rendered stereo display image to fuse at a position different from the surface of the three-dimensional model.) (Xiao [Abstract] The present invention relates to a three-dimensional model reconstruction method and system, the method comprising the following steps: S1, conducting image acquisition on a target using at least one depth camera to obtain a depth image of the target;).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi with Xiao. Obtaining a 3d model and a repaired model and combining, as in Xiao, would benefit the Izumi teachings by allowing a way to combine images or different types of models. Additionally, this is the application of a known technique, 3d fusing models, to yield predictable results.
Regarding claim 14.
Izumi teaches:
A non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs (Izumi [0163] As illustrated in FIG. 26, the information processing apparatus 900 includes CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, RAM (Random Access Memory) 903, and a host bus 904a. In addition, the information processing apparatus 900 includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a coupling port 911, a communication device 913, and a sensor 915. The information processing apparatus 900 may include a processing circuit such as DSP or ASIC in place of or in addition to the CPU 901.):
determining a first intersection-point set between walkthrough light rays corresponding to current walkthrough parameters and the initial three-dimensional model and a second intersection-point set between the walkthrough light rays and the repaired three-dimensional model respectively, wherein the current walkthrough parameters comprise a walkthrough viewing position after moving and a walkthrough viewing angle after moving (Izumi [0098] Here, it looks to the user like the texture at the point P.sub.L and the texture at the point P.sub.R fuse at the position of the point P at which the above-described two light rays intersect, and the surface of the three-dimensional model M32 is present at the position of the point P. [0127] For example, the HMD 3 finds a convenient use when allowing a user to freely walk around within some range. [0090] The positional relationship between the stereo camera and the three-dimensional object corresponds to the positional relationship between the stereo camera and the three-dimensional model in the three-dimensional space. The known positional relationship between the stereo camera and the three-dimensional model in three-dimensional space makes it possible to generate the texture data corresponding to the three-dimensional model as follows from a camera image acquired by the stereo camera. [0099] Moreover, in a case of viewing and listening with the camera positions matching with the positions of the eyes of a user, the light rays at the time of imaging by the cameras are reproduced and the true shape looks reproduced to the user even if the three-dimensional model M32 has any shape. [0107] The data set generation unit 11 generates a plurality of data sets having different data amounts on the basis of the original three-dimensional shape data, the original texture data, the left camera image, and the right camera image as described with reference to FIG. 2. [Figure 14 and Figure 15 [0096] The respective arrows extending from the left camera C31L and the right camera C31R to the point P thus intersect at not the point P, but a point P.sub.R and a point P.sub.L on the surface of the three-dimensional model M32. Distance d between these point P.sub.R and point P.sub.L is mapped as it is as the positional difference between the left-eye texture data and the right-eye texture data, and recorded as it is as left-right parallax. [Figure 13. P and P’ intersection points] [0100] It is not, however, possible to reproduce the actual light rays if the shape of the three-dimensional model M32 includes an error in a case of viewing and listening at positions different from the camera positions. FIG. 13 is a schematic diagram illustrating a viewing and listening situation in which the camera position and the position of an eye of a user are different from each other. In the situation illustrated in FIG. 13, the texture at the point P.sub.L appearing in the left eye E32L of a user and the texture at the point P.sub.L appearing in the right eye E32R of the user fuse at a point P′, and the surface of the three-dimensional model M32 looks present at the position of the point P′. Here, the point P′ illustrated in FIG. 13 is a position different from the point P that is the true tip position of the protrusion. This is because the three-dimensional model M32 includes an error. The point P′, however, looks present at a protruding position as compared with a case where the protrusion looks absent. This suppresses a decrease in the subjective image quality caused by an error even in a case of viewing and listening at positions different from the camera positions.); and
Izumi fails to teach:
acquiring an initial three-dimensional model and a repaired three-dimensional model corresponding to the initial three- dimensional model in a same spatial region, wherein the repaired three-dimensional model is obtained by repairing spatial information in the initial three-dimensional model (Xiao [Pg 1 Par 3] The use of computer technology to model real objects has important significance in various fields. For example, the reconstruction of the three-dimensional model of the human body can reconstruct the posture, motion and topography of the human body in the computer for subsequent Provides a basis for applications such as gesture recognition and replacement of topographical features. [Pg 3 Par 20] In the three-dimensional model reconstruction system of the present invention, the model reconstructor performs surface fusion and hole filling on all the data after registration.);
fusing the initial three-dimensional model and the repaired three-dimensional model according to depth differences between intersection-points of the first intersection-point set and corresponding intersection-points of the second intersection-point set and rendering a fused result to obtain a current walkthrough view (Izumi [0082] As described above, the use of stereo texture data for rendering the texture of a common three-dimensional model allows the rendered stereo display image to fuse at a position different from the surface of the three-dimensional model.) (Xiao [Abstract] The present invention relates to a three-dimensional model reconstruction method and system, the method comprising the following steps: S1, conducting image acquisition on a target using at least one depth camera to obtain a depth image of the target;).
Xiao teaches:
acquiring an initial three-dimensional model and a repaired three-dimensional model corresponding to the initial three- dimensional model in a same spatial region, wherein the repaired three-dimensional model is obtained by repairing spatial information in the initial three-dimensional model (Xiao [Pg 1 Par 3] The use of computer technology to model real objects has important significance in various fields. For example, the reconstruction of the three-dimensional model of the human body can reconstruct the posture, motion and topography of the human body in the computer for subsequent Provides a basis for applications such as gesture recognition and replacement of topographical features. [Pg 3 Par 20] In the three-dimensional model reconstruction system of the present invention, the model reconstructor performs surface fusion and hole filling on all the data after registration.);
Izumi and Xiao teach:
fusing the initial three-dimensional model and the repaired three-dimensional model according to depth differences between intersection-points of the first intersection-point set and corresponding intersection-points of the second intersection-point set and rendering a fused result to obtain a current walkthrough view (Izumi [0082] As described above, the use of stereo texture data for rendering the texture of a common three-dimensional model allows the rendered stereo display image to fuse at a position different from the surface of the three-dimensional model.) (Xiao [Abstract] The present invention relates to a three-dimensional model reconstruction method and system, the method comprising the following steps: S1, conducting image acquisition on a target using at least one depth camera to obtain a depth image of the target;).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi with Xiao. Obtaining a 3d model and a repaired model and combining, as in Xiao, would benefit the Izumi teachings by allowing a way to combine images or different types of models. Additionally, this is the application of a known technique, 3d fusing models, to yield predictable results.
Claim(s) 2-5 and 15-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Izumi et al. (US 20200359007) in view of Xiao et al. (WO 2015188684) and Iwamoto et al. (Iwamoto US 20200349765).
Regarding claim 2.
Izumi and Xiao teach:
The method according to claim 1,
Izumi and Xiao fail to teach:
wherein acquiring the initial three- dimensional model and the repaired three-dimensional model corresponding to the initial three- dimensional model in the same spatial region comprises: generating the initial three-dimensional model according to a panoramic color image and a panoramic depth image in the same spatial region (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.);
and generating the repaired three-dimensional model corresponding to the initial three- dimensional model according to a repaired panoramic color image corresponding to the panoramic color image and a repaired panoramic depth image corresponding to the panoramic depth image (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.).
Iwamoto teaches:
wherein acquiring the initial three- dimensional model and the repaired three-dimensional model corresponding to the initial three- dimensional model in the same spatial region comprises: generating the initial three-dimensional model according to a panoramic color image and a panoramic depth image in the same spatial region (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.);
and generating the repaired three-dimensional model corresponding to the initial three- dimensional model according to a repaired panoramic color image corresponding to the panoramic color image and a repaired panoramic depth image corresponding to the panoramic depth image (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi and Xiao with Iwamoto. Obtaining depth images and panoramic images, as in Iwamoto, would benefit the Izumi and Xiao teachings by allowing a different way of images to be obtained. Additionally, this is the application of a known technique, obtaining depth images and panoramic images, to yield predictable results.
Regarding claim 3.
Izumi, Xiao, and Iwamoto teach:
The method according to claim 2,
wherein before generating the initial three-dimensional model according to the panoramic color image and the panoramic depth image in the same spatial region, the method further comprises: generating the panoramic color image, generating the panoramic depth image, generating the repaired panoramic color image, and generating the repaired panoramic depth image respectively (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi and Xiao with Iwamoto. Obtaining depth images and panoramic images, as in Iwamoto, would benefit the Izumi and Xiao teachings by allowing a different way of images to be obtained. Additionally, this is the application of a known technique, obtaining depth images and panoramic images, to yield predictable results.
Regarding claim 4.
Izumi, Xiao, and Iwamoto teach:
The method according to claim 3,
wherein generating the panoramic depth image comprises: acquiring a plurality of depth images from different viewing angles of shooting in the same spatial region (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.);
and splicing the plurality of depth images to obtain the panoramic depth image (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed. [Figure 6]).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi and Xiao with Iwamoto. Obtaining depth images and panoramic images, as in Iwamoto, would benefit the Izumi and Xiao teachings by allowing a different way of images to be obtained. Additionally, this is the application of a known technique, obtaining depth images and panoramic images, to yield predictable results.
Regarding claim 5.
Izumi, Xiao, and Iwamoto teach:
The method according to claim 4,
wherein splicing the plurality of depth images to obtain the panoramic depth image comprises: splicing the plurality of depth images to obtain the panoramic depth image by using a same splicing method for generating the panoramic color image (Iwamoto [0092] As shown in FIG. 6, after multi-frame 360-degree panoramic scanning is performed on the object, a sequence of depth maps 321 and a sequence of color images 322 are obtained. [Figure 6]).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi and Xiao with Iwamoto. Obtaining depth images and panoramic images, as in Iwamoto, would benefit the Izumi and Xiao teachings by allowing a different way of images to be obtained. Additionally, this is the application of a known technique, obtaining depth images and panoramic images, to yield predictable results.
Regarding claim 15.
Izumi and Xiao teach:
The device according to claim 13,
Izumi and Xiao fail to teach:
wherein acquiring the initial three- dimensional model and the repaired three-dimensional model corresponding to the initial three- dimensional model in the same spatial region comprises: generating the initial three-dimensional model according to a panoramic color image and a panoramic depth image in the same spatial region (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.); and
generating the repaired three-dimensional model corresponding to the initial three- dimensional model according to a repaired panoramic color image corresponding to the panoramic color image and a repaired panoramic depth image corresponding to the panoramic depth image (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.).
Iwamoto teaches:
wherein acquiring the initial three- dimensional model and the repaired three-dimensional model corresponding to the initial three- dimensional model in the same spatial region comprises: generating the initial three-dimensional model according to a panoramic color image and a panoramic depth image in the same spatial region (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.); and
generating the repaired three-dimensional model corresponding to the initial three- dimensional model according to a repaired panoramic color image corresponding to the panoramic color image and a repaired panoramic depth image corresponding to the panoramic depth image (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi and Xiao with Iwamoto. Obtaining depth images and panoramic images, as in Iwamoto, would benefit the Izumi and Xiao teachings by allowing a different way of images to be obtained. Additionally, this is the application of a known technique, obtaining depth images and panoramic images, to yield predictable results.
Regarding claim 16.
Izumi, Xiao, and Iwamoto teach:
The device according to claim 15,
wherein before generating the initial three- dimensional model according to the panoramic color image and the panoramic depth image in the same spatial region, the processor, when executing the computer program, performs: generating the panoramic color image, generating the panoramic depth image,
generating the repaired panoramic color image, and generating the repaired panoramic depth image respectively (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi and Xiao with Iwamoto. Obtaining depth images and panoramic images, as in Iwamoto, would benefit the Izumi and Xiao teachings by allowing a different way of images to be obtained. Additionally, this is the application of a known technique, obtaining depth images and panoramic images, to yield predictable results.
Regarding claim 17.
Izumi, Xiao, and Iwamoto teach:
The device according to claim 16,
wherein generating the panoramic depth image comprises: acquiring a plurality of depth images from different viewing angles of shooting in the same spatial region (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed.); and
splicing the plurality of depth images to obtain the panoramic depth image (Iwamoto [0101] Operation 335: Determine whether a preset quantity of keyframes are stored at intervals of a specific angle (for example, but not limited to a preset angle such as 30 degrees, 45 degrees, 60 degrees, and 90 degrees) in three directions: roll/yaw/pitch; if the quantity of stored keyframes is less than the preset quantity (which is subject to whether a panoramic view of the target object is covered), continue to capture a scene (a color image and a depth map), where the terminal instructs the user to perform more scanning; and if the quantity of keyframes is sufficient to cover the panoramic view of the target object, the user is prompted that scanning completes and a following operation may proceed. [Figure 6]).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi and Xiao with Iwamoto. Obtaining depth images and panoramic images, as in Iwamoto, would benefit the Izumi and Xiao teachings by allowing a different way of images to be obtained. Additionally, this is the application of a known technique, obtaining depth images and panoramic images, to yield predictable results.
Regarding claim 18.
Izumi, Xiao, and Iwamoto teach:
The device according to claim 17, wherein splicing the plurality of depth images to obtain the panoramic depth image comprises: splicing the plurality of depth images to obtain the panoramic depth image by using a same splicing method for generating the panoramic color image (Iwamoto [0092] As shown in FIG. 6, after multi-frame 360-degree panoramic scanning is performed on the object, a sequence of depth maps 321 and a sequence of color images 322 are obtained. [Figure 6]).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi and Xiao with Iwamoto. Obtaining depth images and panoramic images, as in Iwamoto, would benefit the Izumi and Xiao teachings by allowing a different way of images to be obtained. Additionally, this is the application of a known technique, obtaining depth images and panoramic images, to yield predictable results.
Claim(s) 6 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Izumi et al. (US 20200359007) in view of Xiao et al. (WO 2015188684), Iwamoto et al. (Iwamoto US 20200349765), and Sreepathihalli et al. (US 20200053257).
Regarding claim 6.
Izumi, Xiao, and Iwamoto teach:
The method according to claim 5,
Izumi, Xiao, and Iwamoto fail to teach:
wherein before splicing the plurality of depth images to obtain the panoramic depth image, the method comprises: performing depth filling and depth enhancement on the plurality of depth images (Sreepathihalli [0037] For example, the image generation logic 40 may generate an enhanced or “zoomed in” image from the depth sensor data, and fill in or provide data points using interpolation techniques (such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, or edge-directed interpolation).).
Sreepathihalli teaches:
wherein before splicing the plurality of depth images to obtain the panoramic depth image, the method comprises: performing depth filling and depth enhancement on the plurality of depth images (Sreepathihalli [0037] For example, the image generation logic 40 may generate an enhanced or “zoomed in” image from the depth sensor data, and fill in or provide data points using interpolation techniques (such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, or edge-directed interpolation).).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi, Xiao, and Iwamoto. Performing depth filling and enhancements, as in Sreepathihalli, would benefit the Izumi, Xiao, and Iwamoto teachings by allowing a different way adjusting images. Additionally, this is the application of a known technique, performing depth filling and enhancements, to yield predictable results.
Regarding claim 19.
Izumi, Xiao, and Iwamoto teach:
The device according to claim 18,
Izumi, Xiao, and Iwamoto fail to teach:
wherein before splicing the plurality of depth images to obtain the panoramic depth image, the method comprises: performing depth filling and depth enhancement on the plurality of depth images (Sreepathihalli [0037] For example, the image generation logic 40 may generate an enhanced or “zoomed in” image from the depth sensor data, and fill in or provide data points using interpolation techniques (such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, or edge-directed interpolation).).
Sreepathihalli teaches:
wherein before splicing the plurality of depth images to obtain the panoramic depth image, the method comprises: performing depth filling and depth enhancement on the plurality of depth images (Sreepathihalli [0037] For example, the image generation logic 40 may generate an enhanced or “zoomed in” image from the depth sensor data, and fill in or provide data points using interpolation techniques (such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, or edge-directed interpolation).).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi, Xiao, and Iwamoto. Performing depth filling and enhancements, as in Sreepathihalli, would benefit the Izumi, Xiao, and Iwamoto teachings by allowing a different way adjusting images. Additionally, this is the application of a known technique, performing depth filling and enhancements, to yield predictable results.
Claim(s) 7 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Izumi et al. (US 20200359007) in view of Xiao et al. (WO 2015188684), Iwamoto et al. (Iwamoto US 20200349765), and Gausebeck et al. (US 20190026958).
Regarding claim 7.
Izumi, Xiao, and Iwamoto teach:
The method according to claim 3,
Izumi, Xiao, and Iwamoto fail to teach:
wherein generating the panoramic depth image comprises: inputting the panoramic color image into a first pre-trained neural network to obtain the panoramic depth image corresponding to the panoramic color image, wherein the first pre- trained neural network is trained based on a sample panoramic color image and a sample panoramic depth image corresponding to the sample panoramic color image (Gausebeck [0033] In one or more implementations, the 3D-from-2D neural network model can include a model that was trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels. [0100] The one or more panorama models 514 can employ a neural network model that has been trained on panoramic images with 3D ground truth data associated therewith.).
Gausebeck teaches:
wherein generating the panoramic depth image comprises: inputting the panoramic color image into a first pre-trained neural network to obtain the panoramic depth image corresponding to the panoramic color image, wherein the first pre- trained neural network is trained based on a sample panoramic color image and a sample panoramic depth image corresponding to the sample panoramic color image (Gausebeck [0033] In one or more implementations, the 3D-from-2D neural network model can include a model that was trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels. [0100] The one or more panorama models 514 can employ a neural network model that has been trained on panoramic images with 3D ground truth data associated therewith.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi, Xiao, and Iwamoto. Training a neural network, as in Gausebeck, would benefit the Izumi, Xiao, and Iwamoto teachings by allowing a way to be training the ai. Additionally, this is the application of a known technique, training a neural network, to yield predictable results.
Regarding claim 20.
Izumi, Xiao, and Iwamoto teach:
The device according to claim 16,
Izumi, Xiao, and Iwamoto fail to teach:
wherein generating the panoramic depth image comprises: inputting the panoramic color image into a first pre-trained neural network to obtain the panoramic depth image corresponding to the panoramic color image, wherein the first pre- trained neural network is trained based on a sample panoramic color image and a sample panoramic depth image corresponding to the sample panoramic color image (Gausebeck [0033] In one or more implementations, the 3D-from-2D neural network model can include a model that was trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels. [0100] The one or more panorama models 514 can employ a neural network model that has been trained on panoramic images with 3D ground truth data associated therewith.).
Gausebeck teaches:
wherein generating the panoramic depth image comprises: inputting the panoramic color image into a first pre-trained neural network to obtain the panoramic depth image corresponding to the panoramic color image, wherein the first pre- trained neural network is trained based on a sample panoramic color image and a sample panoramic depth image corresponding to the sample panoramic color image (Gausebeck [0033] In one or more implementations, the 3D-from-2D neural network model can include a model that was trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels. [0100] The one or more panorama models 514 can employ a neural network model that has been trained on panoramic images with 3D ground truth data associated therewith.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi, Xiao, and Iwamoto. Training a neural network, as in Gausebeck, would benefit the Izumi, Xiao, and Iwamoto teachings by allowing a way to be training the ai. Additionally, this is the application of a known technique, training a neural network, to yield predictable results.
Claim(s) 8 and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Izumi et al. (US 20200359007) in view of Xiao et al. (WO 2015188684), Iwamoto et al. (Iwamoto US 20200349765), and Sumiyoshi et al. (JP 2018066687).
Regarding claim 8.
Izumi, Xiao, and Iwamoto teach:
The method according to claim 3,
Izumi, Xiao, and Iwamoto fail to teach:
wherein generating the repaired panoramic depth image comprises: determining a depth discontinuous edge in the panoramic depth image, wherein a first side of the depth discontinuous edge is depth foreground, and a second side of the depth discontinuous edge is depth background; and performing depth expansion on the depth foreground and the depth background respectively to obtain the repaired panoramic depth image corresponding to the panoramic depth image (Sumiyoshi [Pg 8 Par 8] In the background image, the motion information V of the edge region E located on the foreground side of the depth threshold among the motion information V of the edge region E calculated by the calculation unit 30G is a position corresponding to the edge region E in the captured image 40. It is the image arranged in. In the foreground image, the motion information V of the edge region E located on the background side of the depth threshold among the motion information V of the edge region E calculated by the calculation unit 30G is a position corresponding to the edge region E in the captured image 40. It is the image arranged in..).
Sumiyoshi teaches:
wherein generating the repaired panoramic depth image comprises: determining a depth discontinuous edge in the panoramic depth image, wherein a first side of the depth discontinuous edge is depth foreground, and a second side of the depth discontinuous edge is depth background; and performing depth expansion on the depth foreground and the depth background respectively to obtain the repaired panoramic depth image corresponding to the panoramic depth image (Sumiyoshi [Pg 8 Par 8] In the background image, the motion information V of the edge region E located on the foreground side of the depth threshold among the motion information V of the edge region E calculated by the calculation unit 30G is a position corresponding to the edge region E in the captured image 40. It is the image arranged in. In the foreground image, the motion information V of the edge region E located on the background side of the depth threshold among the motion information V of the edge region E calculated by the calculation unit 30G is a position corresponding to the edge region E in the captured image 40. It is the image arranged in..).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi, Xiao, and Iwamoto. Adjusting the depth background and foreground, as in Sumiyoshi, would benefit the Izumi, Xiao, and Iwamoto teachings by allowing a way to adjust different parts. Additionally, this is the application of a known technique, adjusting the depth background and foreground, to yield predictable results.
Regarding claim 21.
Izumi, Xiao, and Iwamoto fail teach:
The device according to claim 16,
Izumi, Xiao, and Iwamoto fail to teach:
wherein generating the repaired panoramic depth image comprises: determining a depth discontinuous edge in the panoramic depth image, wherein a first side of the depth discontinuous edge is depth foreground, and a second side of the depth discontinuous edge is depth background; and performing depth expansion on the depth foreground and the depth background respectively to obtain the repaired panoramic depth image corresponding to the panoramic depth image (Sumiyoshi [Pg 8 Par 8] In the background image, the motion information V of the edge region E located on the foreground side of the depth threshold among the motion information V of the edge region E calculated by the calculation unit 30G is a position corresponding to the edge region E in the captured image 40. It is the image arranged in. In the foreground image, the motion information V of the edge region E located on the background side of the depth threshold among the motion information V of the edge region E calculated by the calculation unit 30G is a position corresponding to the edge region E in the captured image 40. It is the image arranged in..).
Sumiyoshi teaches:
wherein generating the repaired panoramic depth image comprises: determining a depth discontinuous edge in the panoramic depth image, wherein a first side of the depth discontinuous edge is depth foreground, and a second side of the depth discontinuous edge is depth background; and performing depth expansion on the depth foreground and the depth background respectively to obtain the repaired panoramic depth image corresponding to the panoramic depth image (Sumiyoshi [Pg 8 Par 8] In the background image, the motion information V of the edge region E located on the foreground side of the depth threshold among the motion information V of the edge region E calculated by the calculation unit 30G is a position corresponding to the edge region E in the captured image 40. It is the image arranged in. In the foreground image, the motion information V of the edge region E located on the background side of the depth threshold among the motion information V of the edge region E calculated by the calculation unit 30G is a position corresponding to the edge region E in the captured image 40. It is the image arranged in..).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi, Xiao, and Iwamoto. Adjusting the depth background and foreground, as in Sumiyoshi, would benefit the Izumi, Xiao, and Iwamoto teachings by allowing a way to adjust different parts. Additionally, this is the application of a known technique, adjusting the depth background and foreground, to yield predictable results.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Izumi et al. (US 20200359007) in view of Xiao et al. (WO 2015188684), Iwamoto et al. (Iwamoto US 20200349765), Sumiyoshi et al. (JP 2018066687), and Dokter et al. (US 20200193690).
Regarding claim 9.
Izumi, Xiao, Iwamoto, and Sumiyoshi teach:
The method according to claim 8,
Izumi, Xiao, Iwamoto, and Sumiyoshi fail to teach:
wherein generating the repaired panoramic color image comprises: performing binarization processing on the repaired panoramic depth image to obtain a binarization mask map; and determining the repaired panoramic color image corresponding to the panoramic color image according to the binarization mask map and the panoramic color image (Dokter [0082] At operation 1006, the training recovery light masks are compared with reference masks. A reference mask may include a binary light mask created for the training panoramic image from which the patch used to create the training recovery light mask was created. In some aspects, the reference mask is created using a first SVM classifier to identify small light sources and a second SVM classifier to identify large light sources.) (Sumiyoshi [Pg 1 Par 4] For example, Patent Document 1 discloses that a circular marker is attached to a measurement target reinforcing bar and a white board is installed on the background of the measurement target reinforcing bar. Patent Document 1 discloses that in this state, binarization processing or the like is performed on a captured image obtained by photographing a measurement target reinforcing bar to measure the diameter of the measurement target reinforcing bar.).
Dokter teaches:
wherein generating the repaired panoramic color image comprises: performing binarization processing on the repaired panoramic depth image to obtain a binarization mask map; and determining the repaired panoramic color image corresponding to the panoramic color image according to the binarization mask map and the panoramic color image (Dokter [0082] At operation 1006, the training recovery light masks are compared with reference masks. A reference mask may include a binary light mask created for the training panoramic image from which the patch used to create the training recovery light mask was created. In some aspects, the reference mask is created using a first SVM classifier to identify small light sources and a second SVM classifier to identify large light sources.) (Sumiyoshi [Pg 1 Par 4] For example, Patent Document 1 discloses that a circular marker is attached to a measurement target reinforcing bar and a white board is installed on the background of the measurement target reinforcing bar. Patent Document 1 discloses that in this state, binarization processing or the like is performed on a captured image obtained by photographing a measurement target reinforcing bar to measure the diameter of the measurement target reinforcing bar.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi, Xiao, Iwamoto, and Sumiyoshi. Having a binarization mask map, as in Dokter, would benefit the Izumi, Xiao, Iwamoto, and Sumiyoshi teachings by allowing a mask parts. Additionally, this is the application of a known technique, having a binarization mask map, to yield predictable results.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Izumi et al. (US 20200359007) in view of Xiao et al. (WO 2015188684), Iwamoto et al. (Iwamoto US 20200349765), Sumiyoshi et al. (JP 2018066687), Dokter et al. (US 20200193690), and Gausebeck et al. (Gausebeck US 20190026958).
Regarding claim 10.
Izumi, Xiao, Iwamoto, Sumiyoshi, and Dokter teach:
The method according to claim 9,
Izumi, Xiao, Iwamoto, Sumiyoshi, and Dokter fail to teach:
wherein determining the repaired panoramic color image corresponding to the panoramic color image according to the binarization mask map and the panoramic color image comprises: inputting the binarization mask map and the panoramic color image into a second pre- trained neural network and performing color repair on the panoramic color image through the second pre-trained neural network to obtain the repaired panoramic color image corresponding to the panoramic color image, wherein the second pre-trained neural network is trained based on a sample binarization mask map, a sample panoramic color image, and a sample repaired panoramic color image corresponding to the sample panoramic color image (Gausebeck [0033] In one or more implementations, the 3D-from-2D neural network model can include a model that was trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels. [0100] The one or more panorama models 514 can employ a neural network model that has been trained on panoramic images with 3D ground truth data associated therewith.).
Gausebeck teaches:
wherein determining the repaired panoramic color image corresponding to the panoramic color image according to the binarization mask map and the panoramic color image comprises: inputting the binarization mask map and the panoramic color image into a second pre- trained neural network and performing color repair on the panoramic color image through the second pre-trained neural network to obtain the repaired panoramic color image corresponding to the panoramic color image, wherein the second pre-trained neural network is trained based on a sample binarization mask map, a sample panoramic color image, and a sample repaired panoramic color image corresponding to the sample panoramic color image (Gausebeck [0033] In one or more implementations, the 3D-from-2D neural network model can include a model that was trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels. [0100] The one or more panorama models 514 can employ a neural network model that has been trained on panoramic images with 3D ground truth data associated therewith.).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi, Xiao, Iwamoto, Sumiyoshi, and Dokter with Gausebeck. Training a neural network, as in Gausebeck, would benefit the Izumi, Xiao, Iwamoto, Sumiyoshi, and Dokter teachings by allowing a way to be training the ai. Additionally, this is the application of a known technique, training a neural network, to yield predictable results.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Izumi et al. (US 20200359007) in view of Xiao et al. (WO 2015188684) and Dokter et al. (US 20200193690).
Regarding claim 11.
Izumi and Xiao teach:
The method according to any one of claim 1,
And using all first intersection-points whose depth differences are less than or equal to zero and all second intersection-points whose depth differences are greater than zero as the fused result of the initial three-dimensional model and the repaired three-dimensional model (Izumi [0082] As described above, the use of stereo texture data for rendering the texture of a common three-dimensional model allows the rendered stereo display image to fuse at a position different from the surface of the three-dimensional model.) (Xiao [Abstract] The present invention relates to a three-dimensional model reconstruction method and system, the method comprising the following steps: S1, conducting image acquisition on a target using at least one depth camera to obtain a depth image of the target;).
Izumi and Xiao fail to teach:
wherein fusing the initial three-dimensional model and the repaired three-dimensional model according to the depth differences between the intersection-points of the first intersection-point set and the corresponding intersection-points of the second intersection-point set comprises: calculating depth differences between first intersection-points in the first intersection- point set and corresponding second intersection-points in the second intersection-point set one by one (Dokter [0091] As shown in FIG. 6A, unbounded depth 634 is calculated by determining the difference between the sampling point 614 and the intersection of the first axis 630 and the second primitive extension 624 which extends from second primitive 620.);
Dokter teaches:
wherein fusing the initial three-dimensional model and the repaired three-dimensional model according to the depth differences between the intersection-points of the first intersection-point set and the corresponding intersection-points of the second intersection-point set comprises: calculating depth differences between first intersection-points in the first intersection- point set and corresponding second intersection-points in the second intersection-point set one by one (Dokter [0091] As shown in FIG. 6A, unbounded depth 634 is calculated by determining the difference between the sampling point 614 and the intersection of the first axis 630 and the second primitive extension 624 which extends from second primitive 620.);
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Izumi and Xiao with Dokter. Having a binarization mask map, as in Dokter, would benefit the Izumi and Xiao teachings by allowing a mask parts. Additionally, this is the application of a known technique, having a binarization mask map, to yield predictable results.
Response to Arguments
Applicant's arguments filed 11/13/2025 have been fully considered but they are not persuasive.
Applicant claims “Izumi focuses on mentally performing human visual filling-in based on (i) the left- eve texture data and (ii) the right-eve texture data” and that the claim is in regards to “It is clearly evident that the "mental human visual filling-in" manner disclosed in Izumi is entirely different from the actual 3D models fusing manner specified in claim 1”
However, Izumi does in fact teach actually fusing.
Firstly, Izumi teaches the walkthrough method [0003] Images viewed from different viewpoints are required to reproduce motion parallax because it is necessary to move a viewpoint in accordance with the head position or the like of a user.
Next, the fusing is not done mentally, nowhere in Izumi does it refer to this fusing as “mental” [0082] As described above, the use of stereo texture data for rendering the texture of a common three-dimensional model allows the rendered stereo display image to fuse at a position different from the surface of the three-dimensional model. This is the same as the principle that, for example, it is possible to provide a stereoscopic effect in spite of a flat display surface in a stereoscopic display that allows an image displayed on a flat screen to provide binocular parallax. The present technology suppresses a decrease in the subjective image quality of a display image rendered at a user viewpoint by using the effect that performing rendering by using such stereo texture allows unevenness different from that of the shape of the three-dimensional model to be recognized.
This “fusing” is through the view of an HMD which would need to have the fusing done in the actually HMD.
Therefore, the rejection stans.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DENIS VASILIY MINKO whose telephone number is (571)270-5226. The examiner can normally be reached Monday-Thursday 8:30-6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at 571-272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DENIS VASILIY MINKO/Examiner, Art Unit 2612
/Said Broome/Supervisory Patent Examiner, Art Unit 2612
Prosecution Insights