DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is in response to Application's amendment/response filed on 07/10/2026, which has been entered and made of record.
Claims 1-20 are pending in the application.
Response to Arguments
Applicant’s arguments with respect to claim 1 regarding the newly added limitations of retrieving a depth value and applying an inverse projection matrix have been considered but are moot in view of the new grounds of rejection represented in this Office Action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 8-11, 13, 16-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Davis et al. (US 20080247636 A1) (Hereinafter referred to as Davis) in view of Zhang et al. (US 20230274490 A1) (Hereinafter referred to as Zhang).
Regarding Claim 1, Davis discloses A method for digital image display, the method comprising: (See Abstract, “A method of using a Graphic User Interface (GUI) for interactive virtual inspection of modeled objects. The method includes acquiring a three-dimensional model of a modeled object and displaying a first view of the modeled object for a user to identify locations of interest on a surface of the modeled object visible within the first view.”)
at a computing device, (See [0014], “FIG. 6 illustrates a networked computing enterprise implementing the present inspection system;”)
receiving an enhanced two-dimensional (2D) image comprising an image dataset derived from a digital object representation, the digital object representation including spatial data representing a three-dimensional (3D) model of an object; (See [0017], “According to various aspects of the present invention, a three-dimensional (3D) model of an object of interest also referred to herein as a virtual object, is utilized for inspection and analysis. An operator interacts with the virtual object by examining, manipulating or otherwise evaluating one or more two-dimensional (2D) and/or three-dimensional (3D) views of the virtual object in a graphic environment.” Also see [0024], “a 2D view of the virtual object, such as may be derived from a corresponding 2D image file or the view may be derived from or otherwise generated from the 3D model, etc.”
Lastly see [0027], “As an illustrative example, a range map 126 can be used to map points back and forth between the 3D model 120 and a corresponding image 122. For example, as shown, a first range map 126A provides pixel-by-pixel range data that maps image points in the first image 122A to corresponding locations on the 3D model. . . Moreover, the unique range data corresponding to each pixel of the first image 122A provides surface location and depth dimension information required to establish a one-to-one mapping between its corresponding image pixel and an associated location on the 3D model.”
In this case, the 3D virtual object corresponds to the “digital object representation”. Thus the surface location and depth information derived from the virtual object, along with the 2D view the 3D virtual object would correspond to an “image dataset” and thus an “enhanced two-dimensional (2D) image”. Note that although Davis doesn’t expressly name and group all these elements together into a defined “image dataset”, since Davis does teach each of the components which form an image dataset, then the limitation is still taught. Lastly, note that having a 3D model implies the existence of “spatial data representing a three-dimensional (3D) model of an object”.)
displaying a two-dimensional (2D) display image included in the image dataset, the 2D display image depicting the object; (See [0017] and [0024] teaching a 2D view of the virtual object, this 2D view would correspond to the “the 2D display image”. Also see [0006], “displaying a first view of the modeled object;”)
receiving user input directed at a selected pixel in the 2D display image having image-space coordinates within the 2D display image; (See [0006], “identifying by a user, a location of interest on a surface of the modeled object that is visible within the first view;” Also see [0017], “As such, the operator may identify a location of interest in a particular 2D or 3D view, which may correspond to a specific feature, region, area or other aspect of the virtual object and associate a markup, e.g., annotation, tag, metadata, etc., with the specifically identified location.”
See [0025], “For purposes of illustration, assume that a first image 122, further designated 1 22A, is taken from a first perspective which is schematically illustrated by the first local coordinate space 124A.”)
retrieving a depth value for the selected pixel from a depth map included in the image dataset based on the image-space coordinates of the selected pixel; (See [0027], “Moreover, the unique range data corresponding to each pixel of the first image 122A provides surface location and depth dimension information required to establish a one-to-one mapping between its corresponding image pixel and an associated location on the 3D model.”)
based at least in part on the image-space coordinates of the selected pixel, the depth value for the selected pixel, calculating 3D object-space coordinates of a selected point on the object corresponding to the selected pixel; and (See Davis [0025] teaching first local coordinate space (image-space coordinates). See Davis [0006] and [0017] teaching to identify a location in a 2D view (selected pixel).
See [0028], “For example, each control/feature point picked by the user in a first view can be mapped to associated 3D global coordinates, e.g., by looking up the appropriate mapping in the range data of a corresponding range map.”
Lastly, see [0045], “map image points on a 2D view that were selected by the operator to corresponding global coordinates on the associated 3D model, e.g., using the range maps 126 described above.”
Here, Davis teaches determining 3D global coordinates (calculating 3D object-space coordinates) of the user identified location within the 2D view (a selected point on the object corresponding to the selected pixel) and teaches doing this by using the range map which, Davis in [0027] teaches that the range map has depth information.)
displaying the 3D object-space coordinates of the selected point. (See [0021], “Global coordinate points of the 3D model are designated at 106 that characterize the location of interest of the modeled object that was identified at 104. Also, a markup tag of user-defined information is created at 108 that annotates the location of interest as will be described in greater detail herein. The markup tag is associated with the designated global coordinate points of the 3D model at 110 and the markup tag is conveyed at 112 when viewing either the 3D model or any one of the image files of the image set that has at least one image point that correlates to a corresponding designated global coordinate point of the 3D model.”
Note that although Davis doesn’t explicitly teach to display the calculated global coordinate points, Davis already teaches displaying a markup tag that contains user-defined information at the designated global coordinate of the 3D model. The global coordinate points (3D object-space coordinates of the selected point) are calculated known information and could be included as a part of “user defined information” for the mark-up tag. Thus, in one scenario, the global coordinate points can be displayed in the markup tag.)
However, Davis fails to explicitly disclose displaying a two-dimensional (2D) display image included in the image dataset, the 2D display image depicting the object from a virtual camera location; . . .
based at least in part on the image-space coordinates of the selected pixel, the depth value for the selected pixel, a model view transformation matrix, and a projection matrix included in the image dataset, calculating 3D object-space coordinates of a selected point on the object corresponding to the selected pixel by applying an inverse projection matrix derived from the model view transformation matrix and the projection matrix to the image-space coordinates and the depth value; and
Zhang teaches the 2D display image depicting the object from a virtual camera location; (See [0048], “The 2D coordinate system 400 may represent, for example, the apparent positions of pixels on a 2D screen, as viewed from the perspective of a virtual camera.”)
based at least in part on the image-space coordinates of the selected pixel, the depth value for the selected pixel, a model view transformation matrix, and a projection matrix included in the image dataset, calculating 3D object-space coordinates of a selected point on the object corresponding to the selected pixel by applying an inverse projection matrix derived from the model view transformation matrix and the projection matrix to the image-space coordinates and the depth value; and (See [0060], “The final projection to the viewport space 630 creates a 2D image to be displayed on the screen (e.g., display 130), with the x, y locations of the normalized device space being preserved and the z locations being applied to sort (e.g., resize) the objects to give the appearance of depth on the 2D image.”
See Fig. 6A showing an example of rendering pipeline for a game engine which shows a process for rendering a 3D scene to a 2D screen. This includes a model matrix 604, view matrix 610, projection matrix 616, etc.
Also see [0061], “Inverses processes of aspects of the rendering pipeline 600 may be performed in order to perform certain actions, such as mapping user input device pointer (e.g., mouse pointer) location to a particular object, and may also be used to generate MV0s, as will be explained below. For example, an inverse view projection matrix 624 may be obtained by multiplying an inverse projection matrix 618 (applied to transform the homogeneous clip space 620 back to the eye space 614) by an inverse view matrix 612 (applied to transform the eye space 614 back to the world space 608). Further, an inverse model matrix 606 may be applied to transform objects in the world space 608 to the local space 602.”
In summary, Zhang teaches x, y (image-space coordinates) and z (the depth value) of a pixel and that one can use the inverse processes in the rendering pipeline to perform actions like obtaining the location of a particular object that a mouse pointer selects (calculating 3D object-space coordinates of a selected point on the object). This is done by applying an inverse projection matrix, as well as using the inverse view and model matrix (a model view transformation matrix). These processes are all shown with connections to each other by the diagram in Fig. 6A.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Davis with Zhang to include using a virtual camera and applying an inverse projection matrix.
The motivation to combine Davis with Zhang would have been obvious as both arts are related to deriving 3D coordinates from a 2D scene (See Zhang [0061]). Although the main way that Davis derives the 3D coordinates is by way of looking it up in a range map, Davis does propose the possibly of using other techniques related to a rendering pipeline to achieve this goal. See Davis [0028], “. . . by looking up the appropriate mapping in the range data of a corresponding range map. Alternatively, if the user is operating with a 3D view, the 3D global coordinates of the picked control/feature point can be extracted from the 3D model such as by using visualization and interaction functions, e.g., using OpenGL or other video standards or by using other techniques.” Note that Davis does mention that this is done while operating in a 3D view, nevertheless, it does point to the usage of a rendering pipeline for deriving 3D coordinates. This contains similarity to Zhang which teaches using the inverse processes in the rendering pipeline to derive the location of a particular object at a mouse pointer input. Thus, a benefit of combining Davis with Zhang is that it can further the applicability of this alternative way of getting 3D coordinates. That being, it can allow for one to derive 3D coordinates while operating in 2D image space instead of needing to operate with a 3D view.
Regarding Claim 2, Davis in view of Zhang disclose The method of claim 1, wherein the depth map includes a plurality of depth values corresponding to a plurality of pixels in the 2D display image. (See Davis [0027], “For example, as shown, a first range map 126A provides pixel-by-pixel range data that maps image points in the first image 122A to corresponding locations on the 3D model. . . Moreover, the unique range data corresponding to each pixel of the first image 122A provides surface location and depth dimension information required to establish a one-to-one mapping between its corresponding image pixel and an associated location on the 3D model.” Note that the range map would correspond to “depth map”.)
Regarding Claim 3, Davis in view of Zhang disclose The method of claim 1, further comprising receiving second user input directed at a second selected pixel in the 2D display image, calculating second 3D object-space coordinates of a second selected point on the object corresponding to the second selected pixel, and displaying the second 3D object-space coordinates of the second selected point. (See Davis [0027], [0028], and [0045] teaching for a user to identify a location in a 2D view, and mapping the point picked by the user to the associated 3D global coordinates (3D object-space coordinates). Also see Davis [0021] teaching to display a markup tag at the calculated 3D global coordinates (displaying the 3D object-space coordinates). Note that since Davis in view of Zhang already teaches doing the above limitations once, then repeating the process for a different identified location would obviously be possible, and the subsequent second 3D object-space coordinates would be calculated and displayed.)
Regarding Claim 8, Davis in view of Zhang disclose The method of claim 1, wherein the image dataset further includes a second model view transformation matrix corresponding to a second coordinate system, and wherein the method further comprises calculating second 3D coordinates of the selected point on the object relative to the second coordinate system based on the second model view transformation matrix. (See Davis [0025], “For purposes of illustration, assume that a first image 122, further designated 1 22A, is taken from a first perspective which is schematically illustrated by the first local coordinate space 124A. A second image, further designated 122B, is taken from a second perspective which is schematically illustrated by the second local coordinate space 124B.” Here, Davis teaches a second viewpoint.
See Zhang Fig. 6A showing a model matrix 604, view matrix 610, projection matrix 616, etc. In combination with Davis, since there is a second viewpoint which is different from a first viewpoint, then implicitly, there should exist a second model / view matrix corresponding to a second coordinate system, which would be used to calculate second 3D coordinates relative to the second coordinate system. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 9, Davis in view of Zhang disclose The method of claim 1, further comprising displaying text metadata associated with the object. (See Davis [0030], “According to an aspect of the present invention, markup tags, i.e., metadata, annotations, etc. that are created with regard to a location of interest on the 3D model may be conveyed to any one or more corresponding 2D view(s).”)
Regarding Claim 10, Davis in view of Zhang disclose The method of claim 1, wherein the computing device is a client computing device, and wherein the enhanced 2D image comprising the image dataset is received from a server computing device via a computer network. (See Davis [0094], “For example, as shown, an enterprise 300 includes a plurality of local client processing systems 302 that may be used to execute an instance of an inspection system according to various aspects of the present invention. . . For example, as shown, the client processing systems 302 communicate over a network 310 to a server 312.” In this case, the local client processing systems 302 corresponds “a client computing device”. Note, although not explicitly taught, receiving the enhanced 2D image comprising the image dataset from a server via a network would be trivial and obvious. A common example and scenario would be a user downloading the enhanced 2D image comprising the image dataset from some server through the network.)
Regarding Claim 11, Davis in view of Zhang disclose The method of claim 10, wherein the 2D display image is displayed by a web browser application of the computing device. (See Davis [0040], “Referring to FIG. 3, various aspects of the present invention may be implemented in a graphics user interface (GUI) 150 of an inspection system that allows an operator to interact with various views of information corresponding to an image set to visually inspect virtual objects.”
Also see Davis [0047], “The functions of each of the above identified modules may be also be changed, combined, simplified, expanded upon or otherwise varied based upon implementation and design requirements for a specific application.” Here, Davis teaches a GUI implementation and although Davis doesn’t explicitly describe the GUI being displayed by specifically a “web browser application”, it would be trivial/obvious to implement for a person ordinarily skilled in the art as a “specific application” would broadly include a commonly used application like a web browser.)
Regarding Claim 13, Davis in view of Zhang disclose A method for enhanced two-dimensional (2D) image creation, the method comprising: (See Davis Abstract, “A method of using a Graphic User Interface (GUI) for interactive virtual inspection of modeled objects.”)
receiving a digital object representation that includes spatial data representing a three-dimensional (3D) model of an object; (See Davis [0006], “acquiring a three-dimensional model of a modeled object;” Note that a 3D model of an object would implicitly contain spatial data.)
rendering a 2D display image of the digital object representation from a virtual camera location; (See Davis [0006], “displaying a first view of the modeled object;”
See Zhang [0048], “The 2D coordinate system 400 may represent, for example, the apparent positions of pixels on a 2D screen, as viewed from the perspective of a virtual camera.”)
generating a depth map including a plurality of depth values corresponding to a plurality of pixels in the 2D display image; and (See Davis [0027], “For example, as shown, a first range map 126A provides pixel-by-pixel range data that maps image points in the first image 122A to corresponding locations on the 3D model. . . Moreover, the unique range data corresponding to each pixel of the first image 122A provides surface location and depth dimension information required to establish a one-to-one mapping between its corresponding image pixel and an associated location on the 3D model.” In this case, a range map corresponds to “a depth map”.)
generating an enhanced 2D image comprising an image dataset for the digital object representation, the image dataset including the 2D display image, the depth map, a model view transformation matrix, and a projection matrix associated with the virtual camera location, (See Davis [0017], “evaluating one or more two-dimensional (2D) and/or three-dimensional (3D) views of the virtual object in a graphic environment.” Also see Davis [0024], “a 2D view of the virtual object, such as may be derived from a corresponding 2D image file or the view may be derived from or otherwise generated from the 3D model, etc.” Also see Davis [0027], “For example, as shown, a first range map 126A provides pixel-by-pixel range data that maps image points in the first image 122A to corresponding locations on the 3D model. . . Moreover, the unique range data corresponding to each pixel of the first image 122A provides surface location and depth dimension information required to establish a one-to-one mapping between its corresponding image pixel and an associated location on the 3D model”
See Zhang Fig. 6A showing a model matrix 604, view matrix 610, projection matrix 616, etc.)
wherein the image dataset is useable to calculate 3D object-space coordinates corresponding to a selected pixel in the 2D display image by applying an inverse projection matrix derived from the model view transformation matrix and the projection matrix to image-space coordinates of the selected pixel and a corresponding depth value in the depth map. (See Zhang [0060], “The final projection to the viewport space 630 creates a 2D image to be displayed on the screen (e.g., display 130), with the x, y locations of the normalized device space being preserved and the z locations being applied to sort (e.g., resize) the objects to give the appearance of depth on the 2D image.”
See Zhang [0061], “Inverses processes of aspects of the rendering pipeline 600 may be performed in order to perform certain actions, such as mapping user input device pointer (e.g., mouse pointer) location to a particular object, and may also be used to generate MV0s, as will be explained below. For example, an inverse view projection matrix 624 may be obtained by multiplying an inverse projection matrix 618 (applied to transform the homogeneous clip space 620 back to the eye space 614) by an inverse view matrix 612 (applied to transform the eye space 614 back to the world space 608). Further, an inverse model matrix 606 may be applied to transform objects in the world space 608 to the local space 602.” The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 16, Claim 16 contains similar limitations as to Claim 8 and is therefore rejected under as similar rationale as that of Claim 8.
Regarding Claim 17, Claim 17 contains similar limitations as to Claim 9 and is therefore rejected under as similar rationale as that of Claim 9.
Regarding Claim 19, Davis in view of Zhang disclose A computing system, comprising: a logic subsystem; and a storage subsystem holding instructions executable by the logic subsystem to: (See Davis [0014], “FIG. 6 illustrates a networked computing enterprise implementing the present inspection system;” See Davis [0007], “In accordance with another aspect of the invention, a system for interactive virtual inspection of modeled objects is provided comprising a processor, a display device coupled to the processor, a memory device”)
receive an enhanced 2D image comprising an image dataset derived from a digital object representation, the digital object representation including spatial data representing a three-dimensional (3D) model of an object; display a 2D display image included in the image dataset via a computer display, the 2D display image depicting the object from a virtual camera location; receive user input directed at a selected pixel in the 2D display image having image-space coordinates within the 2D display image; retrieve a depth value for the selected pixel from a depth map included in the image dataset based on the image-space coordinates of the selected pixel; based at least in part on the image-space coordinates of the selected pixel, the depth value for the selected pixel included in the depth map of the image dataset, a model view transformation matrix, and a projection matrix included in the image dataset, calculate 3D object-space coordinates of a selected point on the object corresponding to the selected pixel by applying an inverse projection matrix derived from the model view transformation matrix and the projection matrix to the image-space coordinates and the depth value; and display the 3D object-space coordinates of the selected point via the computer display. (The above limitations are similar to those of Claim 1 and is therefore rejected under a similar rationale as that of Claim 1)
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Davis in view of Zhang and in further view of Katz et al. (US 20260004440 A1) (Hereinafter referred to as Katz).
Regarding Claim 4, Davis in view of Zhang fails to explicitly disclose The method of claim 3, further comprising calculating a 3D object-space distance between the selected point and the second selected point, and displaying the 3D object-space distance.
Katz teaches calculating a 3D object-space distance between the selected point and the second selected point, and displaying the 3D object-space distance. (See [0005], “determining an alignment between a two-dimensional (2D) image of a dental site and a three-dimensional (3D) surface of the dental site; determining a first point and a second point in the 2D image; projecting the first point and the second point onto the 3D surface based on the alignment between the 2D image and the 3D surface; performing a measurement of a distance between the projected first point and the projected second point on the 3D surface; and displaying at least one of a) a first visualization of the measurement on the 2D image or b) a second visualization of the measurement on the 3D surface.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Davis in view of Zhang with Katz to include calculating a 3D object-space distance between the selected point and the second selected point, and displaying the 3D object-space distance.
The motivation to combine Davis in view of Zhang with Katz would have been obvious as both Davis, Zhang, and Katz are within the same field of determining the coordinates of 2D points within 3D space (See Katz Abstract). Since Davis already teaches being able to calculate the 3D coordinates, and would thus imply having the ability to have calculate two 3D coordinate by repeating the process. Thus, finding the distance between them would simply be applying the distance formula between the two coordinates, which Katz teaches.
Claims 5-7, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Davis in view of Zhang and in further view of Maschmeyer et al. (US 20250069318 A1) (Hereinafter referred to as Maschmeyer).
Regarding Claim 5, Davis in view of Zhang fails to explicitly disclose The method of claim 1, wherein the object includes two or more subcomponents, and wherein the image dataset further comprises a component identity map useable to resolve each pixel of the 2D display image to a specific subcomponent of the object.
Maschmeyer teaches wherein the object includes two or more subcomponents, and (See Fig. 6A showing a shoe object with two or more subcomponents. Each subcomponent being separated by texture.)
wherein the image dataset further comprises a component identity map useable to resolve each pixel of the 2D display image to a specific subcomponent of the object. (See [0071], “An output of the segmentation may comprise a pixel-wise segmentation map of the input image, with each pixel of the image being assigned to a specific class/label. . . As another example, the segmentation output may indicate portions of the texture map that are part of the same object (or object part, sub-part, etc.) as the second point. That is, segmentation output identifying portions of the texture map that are classified into the same object class may be generated.” Here Maschmeyer teaches a pixel-wise segmentation map (component identity map) that is usable to resolve each pixel of the 2D display image to a specific class/label (subcomponent) of the object.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Davis in view of Zhang with Maschmeyer to include objects having two or more subcomponents and a component identity map.
The motivation to combine Davis in view of Zhang with Maschmeyer would have been obvious as both Davis, Zhang, and Maschmeyer are within the same field of displaying virtual objects (See Maschmeyer [0044]) and mapping 3D coordinates to 2D coordinates (See Maschmeyer [0033]). The benefit of having the virtual object have subcomponents and a component identity map would have been to better model real-world objects which commonly contain more than one component.
Regarding Claim 6, Davis in view of Zhang and Maschmeyer disclose The method of claim 5, further comprising determining, based on the component identity map, a selected subcomponent of the two or more subcomponents that is represented by the selected pixel, and displaying a corresponding identifier of the selected subcomponent. (See Maschmeyer [0071], “a pixel-wise segmentation map of the input image, with each pixel of the image being assigned to a specific class/label. . .”
Also see Maschmeyer [0069], “The segmentation model may be provided with a set of defined pixel categories. The categories may be defined, for example, based on user input for specifying one or more categories to which pixels of the input image should be assigned, such as object classes, element classes, sub-element classes, texture classes, etc. The number, types, and identifiers of the categories may be customized by users for classifying the content of the input image.” The motivation to combine would have been similar to that of Claim 5 rejection motivation.)
Regarding Claim 7, Davis in view of Zhang and Maschmeyer disclose The method of claim 6, further comprising, after determining the selected subcomponent, changing color values of each pixel in the 2D display image depicting the selected subcomponent to highlight the selected subcomponent within the 2D display image. (See Maschmeyer [0037], “The graphical representation may facilitate user selection of and changes to identifiable portions of an object's texture.” Also see Maschmeyer [0046], “Texture, such as high frequency detail, surface texture, or color, may be applied to surfaces of a 3D model.” In this case, Maschmeyer teaches being able to change the texture of a user selected portion of an object, in which Maschmeyer’s term of “texture” includes color.
See Maschmeyer [0071], “When a user selects the first point on the 3D model, the display engine may automatically run the segmentation algorithm on the corresponding texture map. A segmentation output identifying portions of the texture map that are classified into the same texture class (e.g., pattern, color, material, etc.) as the second point may be generated.” Also see Maschmeyer [0074], “In particular, the first point may be part of a user request to select similar elements to the first point on the 3D model or to effect defined change(s) to such similar elements.” The motivation to combine would have been similar to that of Claim 5 rejection motivation.)
Regarding Claim 14, Claim 14 contains similar limitations as to Claim 5 and is therefore rejected under as similar rationale as that of Claim 5.
Regarding Claim 20, Claim 20 contains similar limitations as to Claim 5 and is therefore rejected under as similar rationale as that of Claim 5.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Davis in view of Zhang and Maschmeyer and in further view of Nilesh (“The Beginner’s Guide to Semantic Segmentation”).
Regarding Claim 15, Davis in view of Zhang and Maschmeyer fails to explicitly disclose The method of claim 14, wherein the component identity map is generated by generating a second 2D image in which individual subcomponents are assigned a specific class/label. (See Maschmeyer [0071] teaching a pixel-wise segmentation map (component identity map) that is usable to resolve each pixel of the 2D display image to a specific class/label (subcomponent) of the object.)
However, Davis in view of Zhang and Maschmeyer fails to explicitly disclose wherein the component identity map is generated by generating a second 2D image in which individual subcomponents are assigned different unique colors, and then rendered with only ambient lighting enabled.
Nilesh teaches wherein the component identity map is generated by generating a second 2D image in which individual subcomponents are assigned different unique colors, and then rendered with only ambient lighting enabled. (See Nilesh Page 5 showing a pixel-wise segmentation map (2D image) in which different parts are assigned different unique colors and are render with only ambient lighting enabled.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Davis in view of Zhang and Maschmeyer with Nilesh to include assigning each subcomponent different unique colors, and then rendering with only ambient lighting.
The motivation to combine Davis in view of Zhang and Maschmeyer with Nilesh would have been obvious as both Maschmeyer and Nilesh are related to segmentation maps. Nilesh simply shows that it is common in the field of the art for segmentation maps to assigned each portion a unique color. The benefit is that it would be easier to visually distinguish between different portion of the object when they have a unique color.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Davis in view of Zhang and in further view of Troy (US 20220343595 A1).
Regarding Claim 12, Davis in view of Zhang fails to explicitly disclose The method of claim 1, wherein the object represents an aircraft part, and wherein the method further comprises, after receiving the user input directed at the selected pixel, displaying part-specific information relating to the aircraft part, including a part identifier.
Troy teaches wherein the object represents an aircraft part (See [0006], “FIGS. 1A-1C show different example aspects of an aircraft that may be modeled in a three-dimensional (3D) virtual environment.”)
and wherein the method further comprises, after receiving the user input directed at the selected pixel, displaying part-specific information relating to the aircraft part, including a part identifier. (See [0023], “For example, the virtual camera 104 may be positioned to view any 3D modeled component of the aircraft 102 that is part of the 3D virtual environment including, but not limited to, wings, flaps, turbines, doors, windows, landing gear, and/or the tail among other 3D modeled components of the aircraft 102.”
See Davis [0021], “Also, a markup tag of user-defined information is created at 108 that annotates the location of interest as will be described in greater detail herein. The markup tag is associated with the designated global coordinate points of the 3D model at 110 and the markup tag is conveyed at 112 when viewing either the 3D model or any one of the image files of the image set that has at least one image point that correlates to a corresponding designated global coordinate point of the 3D model.” The combination of Davis and Troy would have the markup tags of user-defined information be related to the part-specific information for the aircraft parts.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Davis in view of Zhang with Troy to include have an aircraft as the virtual object and including part-specific information depending on the user selection.
The motivation to combine Davis in view of Zhang with Troy would have been obvious as both Davis and Troy are within the same field of displaying 3D objects and presenting information to the user related to the object (See Troy [0023]). Note that the specification of the object being an aircraft is a trivial implementation, and that displaying part related information would also be obvious to implement as Davis already suggest markup tags that annotates the location of interest, and part information would reasonably be included within the markup tag information. The benefit of including aircraft part-specific information would be that it can quickly present information to the user which can be useful for those working on aircrafts.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Davis in view of Zhang and in further view of Mildrew et al. (US 20180143756 A1) (Hereinafter referred to as Mildrew).
Regarding Claim 18, Davis in view of Zhang fail to explicitly disclose The method of claim 17, wherein the text metadata includes a hyperlink.
Mildrew teaches wherein the text metadata includes a hyperlink. (See [0116], “In some embodiments, the rendering component 426 can provide or present a tag in a 2D frame, such as a pop-up display window, iframe, billboard, banner, etc., that is overlaid onto the representation of the 3D model (e.g., display window 202 as shown in representations 200 and 300). . . The color and opacity of the 2D frame and/or the data/metadata (e.g., text, images, video, hyperlinks, etc.) included therein can also vary to facilitate optimal viewing of the representation of the 3D model and the tag.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Davis in view of Zhang with Mildrew to include text metadata that includes a hyperlink.
The motivation to combine Davis in view of Zhang with Mildrew would have been obvious as both Davis and Mildrew relate to displaying tags in a 2D frame (2D view) on a 3D model, along with text metadata (See Mildrew [0116]). Mildrew simply teaches that hyperlinks are a common addition with text metadata. The benefit of hyperlinks is that it can allow a for a link to be embedded, said link can present a useful resource relating to the 3D model.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANG G HUYNH whose telephone number is (571)272-5432. The examiner can normally be reached Mon-Thu 7:30am-4:30pm EST | Fri 7:30am-11:30am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.G.H./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights