DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Coustaud et al. (U.S. Pub. No. 20210327159) in view of Shakib et al. (U.S. Pub. No. 20180144547).
Regarding claim 1, Coustaud discloses an image processing apparatus comprising: a processor (paragraph 2, line(s) 1-2 "medical image processing"; also, FIG. 1; also, paragraph 4, line(s) 2 "interaction apparatus"), wherein the processor is configured to: display, on a screen (paragraph 13, line(s) 4-6 "processor to at least: generate a virtual environment for display of image content"; also, para 19, “The images can include 2D images, such as an X-ray or CT scan, that can now be rotated and magnified in the VR environment.”), plurality of portions of the three-dimensional image, and the three-dimensional image in a state in which the plurality of two-dimensional images and the three-dimensional image are comparable with each other (para 5, “two-dimensional image slice overlaid on a three-dimensional image volume”; also, para 45, “A control 340 allows a 2D cut or slice 310 of the 3D image volume 320 to be selected”; also, para 45, “FIGS. 3-4, the 2D image slice 310 and the 3D image volume 320 can be of the same”) select a two-dimensional image of interest from among the plurality of two-dimensional images in response to a given selection instruction (paragraph 42, line(s) 7-10 "user can select a slice from within a displayed 3D volume to focus operations, 2D slice views, combination between the 2D slice and the 3D volume, etc."; also, paragraph 45, line(s) 10-11 "A control 340 allows a 2D cut or slice 310 of the 3D image volume 320 to be selected"); and display, on the screen, a portion of interest corresponding to the two-dimensional image of interest among the plurality of portions in a state in which the portion of interest is visually specifiable (FIG. 8-10; also, paragraph 47, line(s) 7-9 "the 2D image slice 810 can be viewed with respect to the 3D volume 820 as well as separate from the 3D volume 820"). Coustaud does not disclose a plurality of two-dimensional images that are obtained by imaging a target object from a plurality of imaging positions and used to generate a three-dimensional image showing the target object in a real space.
However, in a similar field of endeavor, Shakib discloses a plurality of two-dimensional images that are obtained by imaging a target object from a plurality of imaging positions and used to generate a three-dimensional image showing the target object in a real space (para 29, “Using the mobile device camera, a user can walk around a room and capture various images of the room while holding the camera at different positions and orientations relative to the room and the objects in the room. For example, the user can walk around the perimeter of the room and capture different perspectives of the room from different points along the perimeter”; also, para 41, “The 3D model generation component 406 can be configured to employ the sets of 3D data and the 2D image data to generate a 3D model of the real world object or environment. For example, the 3D model generation component 406 can employ an alignment process to generate the 3D model.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Coustaud's invention of displaying a plurality of two-dimensional images in association with portions of a three-dimensional image and in a comparable state, and selecting a two-dimensional image of interest, with the features of Shakib's invention of obtaining a plurality of two-dimensional images of a real-world object from a plurality of imaging positions and using those two-dimensional images to generate a three-dimensional image of the object. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Shakib teaches that "rather than viewing an imperfect view of a 3D reconstruction of an object or environment, it is desirable to be able to view the 2D images or 2D panoramas of the object or environment that were used to generate the 3D reconstruction," so that the same two-dimensional images used to generate the three-dimensional image are made available to the user, which is the comparison between two-dimensional images and the three-dimensional image that Coustaud's interface already facilitates. Second, Shakib teaches that "The 3D model generation component 406 can be configured to employ the sets of 3D data and the 2D image data to generate a 3D model of the real world object or environment," providing a known image-based technique for producing the three-dimensional image that Coustaud displays, applied to Coustaud's system to predictable effect. Third, both references address displaying and navigating between two-dimensional images and a three-dimensional model of a real object, so that incorporating Shakib's image-derived three-dimensional generation into Coustaud's comparative display yields the predictable result of permitting the user to compare the source two-dimensional images with the three-dimensional image generated from them.
Regarding claim 2, Coustaud as modified by Shakib discloses the image processing apparatus according to claim 1, wherein the state in which the plurality of two-dimensional images and the three-dimensional image are comparable with each other is a state in which a first region including the plurality of two-dimensional images and a second region including the three-dimensional image are arranged (Coustaud: FIG. 3-4; also, paragraph 45, line(s) 4-5 "a 2D image 310 is displayed with respect to a 3D volume 320"; also, Paragraph 45, line(s) 13 "3D image volume 320, 2D image slice 310").
Regarding claim 3, Coustaud as modified by Shakib discloses the image processing apparatus according to claim 1, wherein the state in which the portion of interest is visually specifiable includes a state in which the portion of interest is distinguishable from remaining portions among the plurality of portions (Coustaud: FIG. 8-10; also, paragraph 47, line(s) 7-9 "the 2D image slice 810 can be viewed with respect to the 3D volume 820 as well as separate from the 3D volume 820").
Regarding claim 4, Coustaud as modified by Shakib discloses the image processing apparatus according to claim 1, wherein the state in which the portion of interest is visually specifiable includes a state in which the two-dimensional image of interest is distinguishable from remaining two-dimensional images among the plurality of two-dimensional images (Coustaud: para 44, “display a part of a segmented anatomy in the virtual environment 200. The segmented MRI information can be combined with CT scanner data. Different scans, different planes, etc., can be adjusted in the virtual environment 200. The user can interact with each individual representation (e.g., anatomic models, MIP views, 2D slice, MR image, CT image, etc.) to adjust thickness, opacity, position, overlay, measurement, annotation, etc., in the virtual environment 200.").
Regarding claim 5, Coustaud as modified by Shakib discloses the image processing apparatus according to claim 1, wherein the processor is configured to: display, on the screen a plurality of position specifying images in which the plurality of imaging positions at which imaging for obtaining the plurality of two-dimensional images is performed are specifiable, in a state in which the plurality of position specifying images and the three-dimensional image are comparable with each other; select an imaging position corresponding to a position specifying image of interest, which is selected from among the plurality of position specifying images, as an imaging position of interest from among the plurality of imaging positions in response to the selection instruction; and select a two-dimensional image obtained by performing the imaging from the imaging position of interest as the two-dimensional image of interest from among the plurality of two-dimensional images.
However, in a similar field of endeavor, Shakib discloses display, on the screen a plurality of position specifying images in which the plurality of imaging positions at which imaging for obtaining the plurality of two-dimensional images is performed are specifiable, in a state in which the plurality of position specifying images and the three-dimensional image are comparable with each other (para 30, “FIG. 3 presents another example 3D model 300 of the environment generated using 2D and 3D image data of the environment captured by a mobile device camera and including trace lines 302 depicting movement of the mobile device camera through the environment during the capture process.”; also, para 30, “the tick marks 304 on the trace lines 302 indicate points at which 2D and/or 3D image data was captured.”); select an imaging position corresponding to a position specifying image of interest, which is selected from among the plurality of position specifying images, as an imaging position of interest from among the plurality of imaging positions in response to the selection instruction (para 50, “selection of a tag icon included in a 3D model or representation of the 3D model can result in generation of a feature view of the point, area or object associated with the tag icon.”); and select a two-dimensional image obtained by performing the imaging from the imaging position of interest as the two-dimensional image of interest from among the plurality of two-dimensional images (para 72, “the target point can be associated with a preselected 2D image that corresponds to a feature view of the target point. The navigation component 408 can thus move the virtual camera to a position and orientation associated with this preselected 2D image and the rendering component 410 can render this preselected 2D image.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Coustaud's invention of displaying two-dimensional images comparably with a three-dimensional image and selecting a two-dimensional image of interest, as combined with Shakib's invention of generating a three-dimensional image from a plurality of two-dimensional images captured from a plurality of imaging positions, with the features of Shakib's invention of displaying position specifying images that indicate the imaging positions at which the two-dimensional images were captured and selecting an imaging position to retrieve the two-dimensional image captured from that position. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Shakib teaches that the three-dimensional model is generated "including trace lines 302 depicting movement of the mobile device camera through the environment during the capture process," where "the tick marks 304 on the trace lines 302 indicate points at which 2D and/or 3D image data was captured," providing a visual indication of the imaging positions that Coustaud's interface does not provide. Second, Shakib teaches that "selection of a tag icon included in a 3D model or representation of the 3D model can result in generation of a feature view of the point, area or object associated with the tag icon," so that selecting an imaging position retrieves the corresponding captured image. Third, applying these position indicators to Coustaud's comparative display would let the user identify and select the specific imaging position of interest and view the two-dimensional image captured from that position, improving the user's ability to relate each two-dimensional image to its location relative to the three-dimensional image.
Regarding claim 6, Coustaud as modified by Shakib discloses the image processing apparatus according to claim 5, plurality of position specifying images and the three-dimensional image are comparable with each other includes a state in which the plurality of position specifying images and the three-dimensional image face each other.
However, in a similar field of endeavor, Shakib discloses wherein the state in which the plurality of position specifying images and the three-dimensional image are comparable with each other includes a state in which the plurality of position specifying images and the three-dimensional image face each other (para 30, “FIG. 3 presents another example 3D model 300 of the environment generated using 2D and 3D image data of the environment captured by a mobile device camera and including trace lines 302 depicting movement of the mobile device camera through the environment during the capture process.”; also, para 98, “Representation 1200 includes a 3D scene showing original scan locations, as indicated by the dots or dashes of the trace lines 1204.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Coustaud's invention of displaying images comparably with a three-dimensional image, as combined with Shakib's invention of generating a three-dimensional image from a plurality of two-dimensional images captured from a plurality of imaging positions, with the features of Shakib's invention of displaying the position specifying images within the three-dimensional model so that the position specifying images and the three-dimensional image are presented together. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Shakib teaches that the three-dimensional model is generated "including trace lines 302 depicting movement of the mobile device camera through the environment during the capture process," so the position specifying images are presented in the same scene as the three-dimensional image. Second, Shakib teaches that "Representation 1200 includes a 3D scene showing original scan locations, as indicated by the dots or dashes of the trace lines 1204," confirming that the imaging-position indicators and the three-dimensional scene are displayed together for the user. Third, presenting the position specifying images together with the three-dimensional image, as taught by Shakib, would allow the user of Coustaud's comparative display to perceive both at once, which is the comparable, facing arrangement recited.
Regarding claim 7, Coustaud as modified by Shakib discloses the image processing apparatus according to claim 5, wherein the state in which the plurality of two-dimensional images and the three-dimensional image are comparable with each other (Coustaud: FIG. 3, 4; also, paragraph 45, line(s) 15-17 "FIGS. 3-4, the 2D image slice 310 and the 3D image volume 320 can be of the same") is a state in which a third region including the plurality of two-dimensional images and a fourth region (Coustaud: paragraph 79, line(s) 1-2 "two or more representations are registered in the virtual environment 200") image showing an aspect in which the plurality of position specifying images and the three-dimensional image face each other are arranged.
However, in a similar field of endeavor, Shakib discloses including an image showing an aspect in which the plurality of position specifying images and the three-dimensional image face each other are arranged (para 98, “Representation 1200 includes a 3D scene showing original scan locations, as indicated by the dots or dashes of the trace lines 1204”; also, para 30, “FIG. 3 presents another example 3D model 300 of the environment generated using 2D and 3D image data of the environment captured by a mobile device camera and including trace lines 302 depicting movement of the mobile device camera through the environment during the capture process.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Coustaud's invention of arranging regions including the two-dimensional images and the three-dimensional image, as combined with Shakib's invention of generating a three-dimensional image from a plurality of two-dimensional images captured from a plurality of imaging positions, with the features of Shakib's invention of presenting an image of the three-dimensional scene that shows the imaging-position indicators together with the three-dimensional image. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Shakib teaches that "Representation 1200 includes a 3D scene showing original scan locations, as indicated by the dots or dashes of the trace lines 1204," which is an image showing the position specifying images together with the three-dimensional scene. Second, Shakib teaches generating the three-dimensional model "including trace lines 302 depicting movement of the mobile device camera through the environment during the capture process," so the imaging-position indicators are rendered as part of the same view. Third, arranging such an image within Coustaud's comparative display would give the user a dedicated region in which the imaging positions and the three-dimensional image are viewed together, as recited.
Regarding claim 8, Coustaud as modified by Shakib discloses the image processing apparatus according to claim 5portion of interest is visually specifiable includes a state in which the position specifying image of interest is distinguishable from remaining position specifying images among the plurality of position specifying images.
However, in a similar field of endeavor, Shakib discloses wherein the state in which the portion of interest is visually specifiable includes a state in which the position specifying image of interest is distinguishable from remaining position specifying images among the plurality of position specifying images (para 50, “selection of a tag icon included in a 3D model or representation of the 3D model can result in generation of a feature view of the point, area or object associated with the tag icon.”; also, para 82, “The navigation component 408 and/or the rendering component 410 can further select the 2D image associated with the highest similarity score as the "best match" and render that 2D image.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Coustaud's invention of visually specifying a portion of interest, as combined with Shakib's invention of generating a three-dimensional image from a plurality of two-dimensional images captured from a plurality of imaging positions, with the features of Shakib's invention of generating a distinct feature view of the position specifying image of interest selected from among the plurality of position specifying images. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Shakib teaches that "selection of a tag icon included in a 3D model or representation of the 3D model can result in generation of a feature view of the point, area or object associated with the tag icon," so the selected position specifying image is presented distinctly from the others. Second, Shakib teaches that the system "can further select the 2D image associated with the highest similarity score as the 'best match' and render that 2D image," distinguishing the image of interest by rendering it. Third, applying this distinct rendering of the selected position specifying image to Coustaud's visually specifiable display would let the user readily identify which imaging position is currently of interest among the plurality of position specifying images.
Regarding claim 9, Coustaud as modified by Shakib discloses the image processing apparatus according to claim 5, wherein the image processing apparatus has a first operation mode in which the plurality of two-dimensional images and the three-dimensional image are displayed on the screen in the state in which the plurality of two-dimensional images and the three-dimensional image are comparable with each other (Coustaud: paragraph 46, line(s) 1-9 " FIG. 5, the 2D slice 310 can be extracted from the 3D volume 320 in the virtual environment 200 to be viewed separately in the virtual environment 200 by the user. The example of FIG. 6 shows that, using the avatar hand 220, a user can maneuver a composite image formed from a PET image 610 overlaid on a CT image 620 in the virtual environment 200. As shown in the example of FIG. 7, tumors 710, 715 can be highlighted in a cut image 720 in the virtual environment 200"; also, para 45, “FIGS. 3-4, the 2D image slice 310 and the 3D image volume 320 can be of the same”), processor is configured to set any operation mode of the first operation mode or the second operation mode in response to a given setting instruction (Coustaud: paragraph 69, line(s) 10-16 "using the virtual hand 220, the user can move the image content closer, farther, angled, rotated, pull a 2D slice out of a 3D volume, put a 2D slice in a 3D volume, apply a tool and/or other menu 230 option to the image content, position a medical implant and/or instrument with respect to an anatomy in the image content, switch to a different slice view, etc., in the virtual environment 200"). Coustaud does not disclose and a second operation mode in which the plurality of position specifying images are displayed on the screen in the state in which the plurality of position specifying images and the three-dimensional image are comparable with each other.
However, in a similar field of endeavor, Shakib discloses and a second operation mode in which the plurality of position specifying images are displayed on the screen in the state in which the plurality of position specifying images and the three-dimensional image are comparable with each other (para 30, “FIG. 3 presents another example 3D model 300 of the environment generated using 2D and 3D image data of the environment captured by a mobile device camera and including trace lines 302 depicting movement of the mobile device camera through the environment during the capture process.”; also, para 30, “the tick marks 304 on the trace lines 302 indicate points at which 2D and/or 3D image data was captured.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Coustaud's invention of displaying the two-dimensional images and the three-dimensional image in a comparable state and switching among views in response to user input, as combined with Shakib's invention of generating a three-dimensional image from a plurality of two-dimensional images captured from a plurality of imaging positions, with the features of Shakib's invention of displaying, together with the three-dimensional image, the position specifying images that indicate the imaging positions. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Shakib teaches generating the three-dimensional model "including trace lines 302 depicting movement of the mobile device camera through the environment during the capture process," providing a display of the position specifying images that Coustaud's first view does not. Second, Shakib teaches that "the tick marks 304 on the trace lines 302 indicate points at which 2D and/or 3D image data was captured," so the imaging positions are displayed comparably with the three-dimensional image. Third, because Coustaud already teaches that the user can "switch to a different slice view" in response to user input, providing Shakib's position-specifying display as a selectable second mode alongside Coustaud's two-dimensional-image mode would give the user the recited choice between displaying the two-dimensional images and displaying the position specifying images.
Regarding claim 10, Coustaud as modified by Shakib discloses the image processing apparatus according to claim 5, wherein three-dimensional image is displayed on the screen from a viewpoint corresponding to the two-dimensional image of interest.
However, in a similar field of endeavor, Coustaud does not disclose the three-dimensional image is displayed on the screen from a viewpoint corresponding to the two-dimensional image of interest (para 32, “the system can be configured to determine a second perspective that corresponds to a capture position and orientation of an available 2D image of the environment and select that 2D image.”; also, para 81, “the navigation component 408 can be configured to position the virtual cameral relative to the 3D model at a position and orientation corresponding to an original capture position and orientation of a 2D image”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Coustaud's invention of displaying the three-dimensional image and selecting a two-dimensional image of interest, as combined with Shakib's invention of generating a three-dimensional image from a plurality of two-dimensional images captured from a plurality of imaging positions, with the features of Shakib's invention of positioning the viewpoint of the three-dimensional model to correspond to the capture position and orientation of a two-dimensional image. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Shakib teaches that "the system can be configured to determine a second perspective that corresponds to a capture position and orientation of an available 2D image of the environment and select that 2D image," tying the viewpoint of the three-dimensional model to the selected two-dimensional image. Second, Shakib teaches that the navigation component can "position the virtual cameral relative to the 3D model at a position and orientation corresponding to an original capture position and orientation of a 2D image," which is displaying the three-dimensional image from a viewpoint corresponding to the two-dimensional image of interest. Third, applying this viewpoint correspondence to Coustaud's display would let the user view the three-dimensional image from the same vantage point at which the selected two-dimensional image was captured, aiding the comparison between the two.
Regarding claim 11, Coustaud discloses an image processing apparatus comprising: a processor (paragraph 2, line(s) 1-2 "medical image processing"; also, FIG. 1; also, paragraph 4, line(s) 2 "apparatus includes at least one processor"), wherein the processor is configured to: display, on a screen, a plurality of two-dimensional images that are (paragraph 13, line(s) 4-6 "processor to at least: generate a virtual environment for display of image content"; also, para 19, “The images can include 2D images, such as an X-ray or CT scan, that can now be rotated and magnified in the VR environment.”) that are associated with a plurality of portions of the three-dimensional image, and the three-dimensional image in a state in which the plurality of two-dimensional images and the three-dimensional image are comparable with each other (para 5, “two-dimensional image slice overlaid on a three-dimensional image volume”; also, para 45, “A control 340 allows a 2D cut or slice 310 of the 3D image volume 320 to be selected”; also, para 45, “FIGS. 3-4, the 2D image slice 310 and the 3D image volume 320 can be of the same”); select a portion of interest from among the plurality of portions in response to a given selection instruction (paragraph 42, line(s) 7-10 "user can select a slice from within a displayed 3D volume to focus operations, 2D slice views, combination between the 2D slice and the 3D volume, etc."; also, para 38, “user can access to the full 3D/4D content to identify a region of interest and/or an object in the image(s) 205, 210, segment the image(s) 205, 210, manipulate the image(s) 205, 210, etc.”); select a two-dimensional image of interest corresponding to the portion of interest from among the plurality of two-dimensional images (paragraph 45, line(s) 10-11 "A control 340 allows a 2D cut or slice 310 of the 3D image volume 320 to be selected"; also, para 42, “user can select a slice from within a displayed 3D volume to focus operations, 2D slice views, combination between the 2D slice and the 3D volume, etc.”); and display, on the screen, the plurality of two-dimensional images in a state in which the two-dimensional image of interest is distinguishable from remaining two-dimensional images among the plurality of two-dimensional images (FIG. 6-7; also, paragraph 44, line(s) 2-10 "display a part of a segmented anatomy in the virtual environment 200. The segmented MRI information can be combined with CT scanner data. Different scans, different planes, etc., can be adjusted in the virtual environment 200. The user can interact with each individual representation (e.g., anatomic models, MIP views, 2D slice, MR image, CT image, etc.) to adjust thickness, opacity, position, overlay, measurement, annotation, etc., in the virtual environment 200."). Coustaud does not disclose obtained by imaging a target object from a plurality of imaging positions and used to generate a three-dimensional image showing the target object in a real space.
However, in a similar field of endeavor, Shakib discloses obtained by imaging a target object from a plurality of imaging positions and used to generate a three-dimensional image showing the target object in a real space (para 29, “Using the mobile device camera, a user can walk around a room and capture various images of the room while holding the camera at different positions and orientations relative to the room and the objects in the room. For example, the user can walk around the perimeter of the room and capture different perspectives of the room from different points along the perimeter.”; also, para 41, “The 3D model generation component 406 can be configured to employ the sets of 3D data and the 2D image data to generate a 3D model of the real world object or environment. For example, the 3D model generation component 406 can employ an alignment process to generate the 3D model.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Coustaud's invention of displaying a plurality of two-dimensional images in association with portions of a three-dimensional image and in a comparable state, selecting a portion of interest, and distinguishably displaying a two-dimensional image of interest corresponding to the portion of interest, with the features of Shakib's invention of obtaining a plurality of two-dimensional images of a real-world object from a plurality of imaging positions and using those two-dimensional images to generate a three-dimensional image of the object. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Shakib teaches that "rather than viewing an imperfect view of a 3D reconstruction of an object or environment, it is desirable to be able to view the 2D images or 2D panoramas of the object or environment that were used to generate the 3D reconstruction," so that the two-dimensional images used to generate the three-dimensional image are themselves available for the comparison Coustaud's interface facilitates. Second, Shakib teaches that "The 3D model generation component 406 can be configured to employ the sets of 3D data and the 2D image data to generate a 3D model of the real world object or environment," providing a known image-based technique for producing the three-dimensional image that Coustaud displays. Third, both references address displaying and navigating between two-dimensional images and a three-dimensional model of a real object, so that incorporating Shakib's image-derived three-dimensional generation into Coustaud's comparative display yields the predictable result of permitting the user to select a portion of the three-dimensional image and distinguishably view the corresponding source two-dimensional image.
Regarding claim 12, Coustaud as modified by Shakib discloses the image processing apparatus according to claim 11, wherein the processor is configured to: display, on the screen, a plurality of position specifying images in which the plurality of imaging positions at which imaging for obtaining the plurality of two-dimensional images is performed are specifiable, in a state in which the plurality of position specifying images and the three-dimensional image are comparable with each other select a position specifying image of interest from among the plurality of position specifying images in response to the selection instruction; and select the two-dimensional image obtained by performing the imaging from the imaging position specified from the position specifying image of interest as the two-dimensional image of interest from among the plurality of two-dimensional images.
However, in a similar field of endeavor, Shakib discloses display, on the screen, a plurality of position specifying images in which the plurality of imaging positions at which imaging for obtaining the plurality of two-dimensional images is performed are specifiable, in a state in which the plurality of position specifying images and the three-dimensional image are comparable with each other (para 30, “FIG. 3 presents another example 3D model 300 of the environment generated using 2D and 3D image data of the environment captured by a mobile device camera and including trace lines 302 depicting movement of the mobile device camera through the environment during the capture process.”; also, para 30, “the tick marks 304 on the trace lines 302 indicate points at which 2D and/or 3D image data was captured.”) select a position specifying image of interest from among the plurality of position specifying images in response to the selection instruction (para 50, “selection of a tag icon included in a 3D model or representation of the 3D model can result in generation of a feature view of the point, area or object associated with the tag icon.”); and select the two-dimensional image obtained by performing the imaging from the imaging position specified from the position specifying image of interest as the two-dimensional image of interest from among the plurality of two-dimensional images (para 72, “the target point can be associated with a preselected 2D image that corresponds to a feature view of the target point. The navigation component 408 can thus move the virtual camera to a position and orientation associated with this preselected 2D image and the rendering component 410 can render this preselected 2D image.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Coustaud's invention of displaying two-dimensional images comparably with a three-dimensional image and selecting a two-dimensional image of interest, as combined with Shakib's invention of generating a three-dimensional image from a plurality of two-dimensional images captured from a plurality of imaging positions, with the features of Shakib's invention of displaying position specifying images that indicate the imaging positions and selecting a position specifying image to retrieve the two-dimensional image captured from the specified imaging position. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Shakib teaches generating the three-dimensional model "including trace lines 302 depicting movement of the mobile device camera through the environment during the capture process," where "the tick marks 304 on the trace lines 302 indicate points at which 2D and/or 3D image data was captured," supplying the position specifying images. Second, Shakib teaches that "selection of a tag icon included in a 3D model or representation of the 3D model can result in generation of a feature view of the point, area or object associated with the tag icon," so a selected position specifying image retrieves its corresponding image. Third, Shakib teaches that "The navigation component 408 can thus move the virtual camera to a position and orientation associated with this preselected 2D image and the rendering component 410 can render this preselected 2D image," so that incorporating these features into Coustaud's comparative display would let the user select an imaging position and view the two-dimensional image captured from that position.
Regarding claim 13, Coustaud discloses an image processing method comprising: displaying, on a screen, a plurality of two-dimensional images (para 19, “Certain examples also provide a method and mechanism to scan and/or to take a previously scanned 3D object, such as a tool, implant, etc., and display it in the VR environment”; also, para 13, “processor to at least: generate a virtual environment for display of image content”; also, para 19, “The images can include 2D images, such as an X-ray or CT scan, that can now be rotated and magnified in the VR environment”) plurality of portions of the three-dimensional image, and the three-dimensional image in a state in which the plurality of two-dimensional images and the three-dimensional image are comparable with each other (para 5, “two-dimensional image slice overlaid on a three-dimensional image volume”; also, para 45, “FIGS. 3-4, the 2D image slice 310 and the 3D image volume 320 can be of the same”); selecting a two-dimensional image of interest from among the plurality of two-dimensional images in response to a given selection instruction (paragraph 42, line(s) 7-10 "user can select a slice from within a displayed 3D volume to focus operations, 2D slice views, combination between the 2D slice and the 3D volume, etc."; also, paragraph 45, line(s) 10-11 "A control 340 allows a 2D cut or slice 310 of the 3D image volume 320 to be selected"); and the three-dimensional image in a state in which the plurality of two-dimensional images and the three-dimensional image are comparable with each other (FIG. 3, 4; also, paragraph 45, line(s) 15-17 "FIGS. 3-4, the 2D image slice 310 and the 3D image volume 320 can be of the same"); selecting a two-dimensional image of interest from among the plurality of two-dimensional images in response to a given selection instruction (paragraph 42, line(s) 7-10 "user can select a slice from within a displayed 3D volume to focus operations, 2D slice views, combination between the 2D slice and the 3D volume, etc."; also, paragraph 45, line(s) 10-11 "A control 340 allows a 2D cut or slice 310 of the 3D image volume 320 to be selected"); and displaying, on the screen, a portion of interest corresponding to the two-dimensional image of interest among the plurality of portions in a state in which the portion of interest is visually specifiable (FIG. 8-10; also, paragraph 47, line(s) 7-9 "the 2D image slice 810 can be viewed with respect to the 3D volume 820 as well as separate from the 3D volume 820"). Coustaud does not disclose obtained by imaging a target object from a plurality of imaging positions and that are used to generate a three-dimensional image showing the target object in a real space.
However, in a similar field of endeavor, Shakib discloses obtained by imaging a target object from a plurality of imaging positions and that are used to generate a three-dimensional image showing the target object in a real space (para 29, “Using the mobile device camera, a user can walk around a room and capture various images of the room while holding the camera at different positions and orientations relative to the room and the objects in the room. For example, the user can walk around the perimeter of the room and capture different perspectives of the room from different points along the perimeter.”; also, para 41, “The 3D model generation component 406 can be configured to employ the sets of 3D data and the 2D image data to generate a 3D model of the real world object or environment. For example, the 3D model generation component 406 can employ an alignment process to generate the 3D model.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Coustaud's invention of displaying a plurality of two-dimensional images in association with portions of a three-dimensional image and in a comparable state, selecting a two-dimensional image of interest, and displaying a corresponding portion of interest, with the features of Shakib's invention of obtaining a plurality of two-dimensional images of a real-world object from a plurality of imaging positions and using those two-dimensional images to generate a three-dimensional image of the object. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Shakib teaches that "rather than viewing an imperfect view of a 3D reconstruction of an object or environment, it is desirable to be able to view the 2D images or 2D panoramas of the object or environment that were used to generate the 3D reconstruction," so that the two-dimensional images used to generate the three-dimensional image are themselves available for the comparison Coustaud's interface facilitates. Second, Shakib teaches that "The 3D model generation component 406 can be configured to employ the sets of 3D data and the 2D image data to generate a 3D model of the real world object or environment," providing a known image-based technique for producing the three-dimensional image that Coustaud displays. Third, both references address displaying and navigating between two-dimensional images and a three-dimensional model of a real object, so that incorporating Shakib's image-derived three-dimensional generation into Coustaud's comparative display yields the predictable result of permitting the user to compare the source two-dimensional images with the three-dimensional image generated from them.
Regarding claim 14, Coustaud discloses a non-transitory computer-readable storage medium storing a program for causing a computer to execute a process displaying, on a screen, a plurality of two-dimensional images (para 14, “non-transitory computer readable storage medium includes instructions which, when executed, cause at least one processor to at least: generate a virtual environment for display of image content “; also, para 19, “The images can include 2D images, such as an X-ray or CT scan, that can now be rotated and magnified in the VR environment. “) associated with a plurality of portions of the three-dimensional image, and the three-dimensional image in a state in which the plurality of two-dimensional images and the three-dimensional image are comparable with each other (para 5, “two-dimensional image slice overlaid on a three-dimensional image volume “, also, FIG. 3, 4; also, paragraph 45, line(s) 15-17 "FIGS. 3-4, the 2D image slice 310 and the 3D image volume 320 can be of the same") selecting a two-dimensional image of interest from among the plurality of two-dimensional images in response to a given selection instruction (paragraph 42, line(s) 7-10 "user can select a slice from within a displayed 3D volume to focus operations, 2D slice views, combination between the 2D slice and the 3D volume, etc."; also, paragraph 45, line(s) 10-11 "A control 340 allows a 2D cut or slice 310 of the 3D image volume 320 to be selected") and displaying, on the screen, a portion of interest corresponding to the two-dimensional image of interest among the plurality of portions in a state in which the portion of interest is visually specifiable (FIG. 8-10; also, paragraph 47, line(s) 7-9 "the 2D image slice 810 can be viewed with respect to the 3D volume 820 as well as separate from the 3D volume 820"). Coustaud does not disclose obtained by imaging a target object from a plurality of imaging positions and that are used to generate a three-dimensional image showing the target object in a real space.
However, in a similar field of endeavor, Shakib discloses obtained by imaging a target object from a plurality of imaging positions and that are used to generate a three-dimensional image showing the target object in a real space (para 29, “Using the mobile device camera, a user can walk around a room and capture various images of the room while holding the camera at different positions and orientations relative to the room and the objects in the room. For example, the user can walk around the perimeter of the room and capture different perspectives of the room from different points along the perimeter.”; also, para 41, “The 3D model generation component 406 can be configured to employ the sets of 3D data and the 2D image data to generate a 3D model of the real world object or environment. For example, the 3D model generation component 406 can employ an alignment process to generate the 3D model.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Coustaud's invention of displaying a plurality of two-dimensional images in association with portions of a three-dimensional image and in a comparable state, selecting a two-dimensional image of interest, and displaying a corresponding portion of interest, with the features of Shakib's invention of obtaining a plurality of two-dimensional images of a real-world object from a plurality of imaging positions and using those two-dimensional images to generate a three-dimensional image of the object. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Shakib teaches that "rather than viewing an imperfect view of a 3D reconstruction of an object or environment, it is desirable to be able to view the 2D images or 2D panoramas of the object or environment that were used to generate the 3D reconstruction," so that the two-dimensional images used to generate the three-dimensional image are themselves available for the comparison Coustaud's interface facilitates. Second, Shakib teaches that "The 3D model generation component 406 can be configured to employ the sets of 3D data and the 2D image data to generate a 3D model of the real world object or environment," providing a known image-based technique for producing the three-dimensional image that Coustaud displays. Third, both references address displaying and navigating between two-dimensional images and a three-dimensional model of a real object, so that incorporating Shakib's image-derived three-dimensional generation into Coustaud's comparative display yields the predictable result of permitting the user to compare the source two-dimensional images with the three-dimensional image generated from them.
Response to Arguments
Applicant's arguments filed 05/07/2026 have been fully considered but they are not persuasive.
Applicant argues (Remarks, pages 9-10) that Coustaud fails to teach the amended feature of independent claim 1 requiring "a plurality of two-dimensional images that are obtained by imaging a target object from a plurality of imaging positions and used to generate a three-dimensional image," contending that Coustaud "only discloses post-reconstruction slice extraction from a 3D dataset" because "the dashed plane 330 represents a cross-sectional position within an already generated 3D volume, and the selected 2D image corresponds to a slice extracted from the 3D image."
Applicant's amendment has necessitated a new ground of rejection. The amended feature directed to two-dimensional images that are obtained by imaging a target object from a plurality of imaging positions and used to generate a three-dimensional image showing the target object in a real space is now addressed under 35 U.S.C. 103 by the combination of Coustaud with Shakib (US 2018/0144547 A1), as set forth below. Shakib expressly teaches capturing a plurality of two-dimensional images of a real-world object from a plurality of different imaging positions and using those two-dimensional images to generate a three-dimensional model of the object. Coustaud teaches the remaining features, namely displaying the plurality of two-dimensional images in association with portions of the three-dimensional image and in a state comparable with the three-dimensional image, selecting a two-dimensional image of interest in response to a selection instruction, and displaying a portion of interest in a visually specifiable state.
To the extent Applicant argues that Coustaud's displayed slices are extracted from an already-existing three-dimensional volume, that argument does not address the combination now applied: the amended generation feature is supplied by Shakib, while Coustaud supplies the comparative display and selection framework. Applicant's argument that dependent claims 2-10 and 12 are allowable solely by virtue of their dependency (Remarks, page 10) is likewise not persuasive, because the independent claims from which they depend stand rejected under the combination set forth.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jai Li whose telephone number is (571)272-1170. The examiner can normally be reached Mon-Thu between 06:00-16:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAI W LI/Junior Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights