Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Busey (US 20210110557 A1), and further in view of Elahie (US 12106437 B1).
Regarding claim 1, Busey teaches a method, comprising:
receiving first information about a three-dimensional (“3D”) surface characterized by a 3D mesh, the 3D mesh comprising a plurality of 3D points (par. 0018: “Some embodiments may analyze visual or other data to determine a partial or entire 2D or 3D reconstruction of the target physical object, such as point-cloud or polygon mesh in the world coordinate system (i.e. “world-space coordinate system”) representing cloth topology, surface roughness, or the like. Some embodiments may infer a virtual representation target physical object (e.g., the point-cloud or polygon mesh in the world coordinate system), and in some cases, the virtual representation may include the contour (or other texture data) and the set of points.”);
receiving second information about a two-dimensional (“2D”) surface, the 2D surface comprising a plurality of 2D points (par. 0061: “Some embodiments may determine attributes of a planar surface based on estimated positions of a cluster of feature points.”);par.
receiving, from a user, a first selection of a set of 2D anchor points from among the plurality of 2D points (par. 0067: “Alternatively, or in addition, a virtual object may be selected by a user to be added to a set of new virtual objects before or during the display of the set of new virtual objects further described below. For example, a model representing a first virtual object may be selected from a user interface and a version of the first virtual object may then be displayed to the user or other users executing different instances of an application having access to the selected virtual object.”);
for each 2D anchor point of the set of 2D anchor points, receiving from the user a second selection of a 3D anchor point mapped to the 2D anchor point to generate a set of 3D anchor points (par. 0071: “The textures may be combined using a UV texture mapping operation, such as forward texture mapping, inverse texture mapping, affine texture mapping, or the like.”);
determining a first mapping of the plurality of 2D points to the plurality of 3D points using the set of 2D anchor points and the set of 3D anchor points, comprising:
selecting a first set of 3D points (par. 0018: “Some embodiments may determine spatial data associated with the target object, such as a set of points (e.g., in world coordinate system or an image coordinate system) representing points on the detected edge of the target physical object, a bounding box of the target physical object, or a contour of the target physical object.”); and
for each 3D point of the first set of 3D points:
computing, using a forward operation, a 2D point mapped to the 3D point based on the first set of weights and the set of 2D anchor points (par. 0071: “The textures may be combined using a UV texture mapping operation, such as forward texture mapping, inverse texture mapping, affine texture mapping, or the like.”); and
outputting a first image of the 2D surface mapped onto the 3D surface using the first mapping (par. 0083: “The sensor outputs of the Lidar sensor may be used to provide environmental information in the form of a three-dimensional depth map of the space surrounding the mobile computing device.”).
Busey fails to teach:
for each 3D point of the first set of 3D points:
computing, using an inverse operation, a first set of weights, the first set of weights comprising a respective weight for each 3D anchor point of the set of 3D anchor points such that the weighted average of the set of 3D anchor points using the first set of weights corresponds to the 3D point; and
Elahie teaches a method, comprising:
receiving first information about a three-dimensional (“3D”) surface characterized by a 3D mesh, the 3D mesh comprising a plurality of 3D points (col. 1, line 66 – col. 2, line 5: “Provided are systems and associated methods for generating point clouds with infinitely scalable resolutions from a three-dimensional (3D) mesh model. The systems and associated methods convert the individual polygons of the 3D mesh model into any number of discrete points, and generate a point cloud representation for the object or scene of the 3D mesh model based on the generated points.”);
receiving second information about a two-dimensional (“2D”) surface, the 2D surface comprising a plurality of 2D points (col. 3, lines 19-25: “The desired distribution specifies the number of points to represent the flat plane and the relative positioning for the points. Accordingly, each point is defined with coordinates that specify a position in 3D space that is between the coordinates of the polygon vertices and along the flat plane that connects the vertices.”);
for each 2D anchor point of the set of 2D anchor points, receiving from the user a second selection of a 3D anchor point mapped to the 2D anchor point to generate a set of 3D anchor points (col. 8, lines 27-35: “Process 500 includes determining (at 506) a distribution for the points that are to replace the polygons of the 3D mesh model. The distribution may be defined as a specific resolution, a desired density of points in a particular region of 3D space, a number of points to be generated for each polygon or relative to an amount of space spanned by each polygon, and/or other criteria. The distribution may be defined by user input or may be a configurable setting of 3D modeling system 100.”);
determining a first mapping of the plurality of 2D points to the plurality of 3D points using the set of 2D anchor points and the set of 3D anchor points (col. 8, lines 59-65: “In some embodiments, 3D modeling system 100 may plot the distribution over the flat plane of the particular polygon, determine a relative mapping for the point positions in the distribution on the flat plane, and determine the weightings to the vertex coordinates that correspond to the point positions and specify the coordinate values for the points.”), comprising:
selecting a first set of 3D points (col. 8, lines 45-49: “Process 500 includes generating (at 510) a different set of points to replace each selected (at 508) polygon according to the determined (at 506) distribution. Each of the set of points that is generated (at 510) for a particular polygon is defined with a position about the flat plane of the particular polygon.”); and
for each 3D point of the first set of 3D points:
computing, using an inverse operation, a first set of weights, the first set of weights comprising a respective weight for each 3D anchor point of the set of 3D anchor points such that the weighted average of the set of 3D anchor points using the first set of weights corresponds to the 3D point (col. 8, lines 50-55: “The position is derived from the vertex coordinates of the particular polygon. For instance, 3D modeling system 100 may associate different weights to the vertex coordinates of the particular polygon to generate (at 510) the set of points for the particular polygon according to the determined (at 506) distribution.”); and
computing, using a forward operation, a 2D point mapped to the 3D point based on the first set of weights and the set of 2D anchor points (col. 8, lines 59-65, as above); and
outputting a first image of the 2D surface mapped onto the 3D surface using the first mapping (col. 9, lines 57-60: “Process 500 includes presenting (at 514) a 3D visualization on a display by rendering the displaced (at 512) points that are generated (at 510) to replace the polygons of the 3D mesh model.”).
Elahie fails to teach:
receiving, from a user, a first selection of a set of 2D anchor points from among the plurality of 2D points.
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to combine the pattern-triggered object modification of Busey with the point cloud generation method of Elahie, as both are in the same field of endeavor of finite element generation in three-dimensional modelling for computer graphics. Busey does teach inverse texture mapping (par. 0071, as above) as a means of combining textures, but does not specify the operation thereof. Using this operation as Elahie does is therefore an obvious innovation to one familiar in the art.
Regarding claim 2, Busey and Elahie teach the method of claim 1. Busey further teaches determining an embedded representation of the plurality of 3D points (par. 0018: “Some embodiments may analyze visual or other data to determine a partial or entire 2D or 3D reconstruction of the target physical object, such as point-cloud or polygon mesh in the world coordinate system (i.e. “world-space coordinate system”) representing cloth topology, surface roughness, or the like. Some embodiments may infer a virtual representation target physical object (e.g., the point-cloud or polygon mesh in the world coordinate system), and in some cases, the virtual representation may include the contour (or other texture data) and the set of points.”).
Regarding claim 3, Busey and Elahie teach the method of claim 2. Busey further teaches wherein determining the embedded representation of the plurality of 3D points comprises:
determining a subset of the plurality of 3D points (par. 0066: “In some operations, the process 300 includes operations to select a set of virtual objects, as indicated by block 320. A virtual object may include a two-dimensional (2D) object, such as a picture, a texture, a design, or the like. Alternatively, a virtual object may include a 3D object, such as a 3D model, a cloud of points, or other values associated with three-dimensional positions in a space.” NOTE: given “a set of virtual objects,” a single virtual object would logically be a subset. Both are comprised of a plurality of 3D points.);
generating a matrix including a distance between each first 3D point of the plurality of 3D points and each second 3D point of the subset of the plurality of 3D points (par. 0032: “In some embodiments, a hardware tensor accelerator may include integrated circuitry for a matrix multiplier unit, united buffer unit, and activation unit for activation functions. By using specialized integrated circuits for one-dimensional array operations and multi-dimensional matrix multiplication operations, the various low-precision, high-volume computations useful for use during computer vision operations or other augmented reality operations may be improved.”);
computing a first embedded coordinate for each second 3D point of the subset of the plurality of 3D points using the matrix and a metric multi-dimensional scaling technique (par. 0098: “For example, if the position of a target feature is represented by the coordinates [53, 79, 422], some embodiments may set the position of an anchor point at the coordinates [53, 79, 422]. Alternatively, or in addition, an anchor point may be set at a location based on a set of locations of the set of target features.”); and
computing a second embedded coordinate for each first 3D point of the plurality of 3D points using an interpolation technique (par. 0081: “In some embodiments, the second virtual object may be generated based on a modification of a first virtual object. Such modifications may include updates to a design, color, size, mesh geometry, another mesh parameter, deformation geometry, textures, anchor point positions, or the like.”).
Regarding claim 4, Busey and Elahie teach the method of claim 3. Busey further teaches wherein:
the inverse operation comprises computing a distance between the 3D point and each 3D anchor point of the set of 3D anchor points (par. 0098: “in some embodiments, operations of the process 400 may include determining a set of anchor points or a set of virtual objects based on the set of target features and sending the associated data to the mobile computing device, as indicated by block 428. In some embodiments, the position of a target feature with respect to the features of the rest of an image may be the same as the position of the anchor point with respect to the features of the image. For example, if the position of a target feature is represented by the coordinates [53, 79, 422], some embodiments may set the position of an anchor point at the coordinates [53, 79, 422]. Alternatively, or in addition, an anchor point may be set at a location based on a set of locations of the set of target features.”); and
each distance between the 3D point and a 3D anchor point is based on the embedded representation of the 3D point and the 3D anchor point (as above).
Regarding claim 5, Busey and Elahie teach the method of claim 1. Busey further teaches wherein:
the 3D mesh is comprised of a plurality of 2D triangles, each 2D triangle being defined by three 3D vertices from the plurality of 3D points (par. 0060: “For example, some embodiments may generate or otherwise update a representation of a physical object that includes a surface model of the physical object, a rigged mesh model of the physical object including movable points indicating where the model can deform or otherwise move, colors of the physical object, surface contours of the physical object, or the like.”); and
each 3D anchor point of the set of 3D anchor points is a point inside a 2D triangle of the plurality of 2D triangles, wherein each 3D anchor point corresponds to three 3D points of the plurality of 3D points that are the three 3D vertices of the 2D triangle (par. 0063: “Some embodiments may assign a position on the surface of a virtual representation of a physical object or inside of the virtual representation of the physical object as an anchor point. For example, some embodiments may select a position on a planar surface in virtual space that is representing a table in physical space as an anchor point, to which one or more virtual objects may be attached to in a virtual space. Some embodiments may include instructions to re-rig a mesh of the virtual representation based on additional visual information. For example, some embodiments may detect a pivoting motion of an article of clothing (or another object) at a first position not represented by a point on a rigged mesh.”).
Regarding claim 6, Busey and Elahie teach the method of claim 5. Busey further teaches wherein the first set of 3D points comprises a set of 3D vertices corresponding to the plurality of 2D triangles associated with each 3D anchor point of the set of 3D anchor points (par. 0063, as above in claim 5 rejection).
Regarding claim 7, Busey and Elahie teach the method of claim 5. Busey further teaches wherein determining the first mapping further comprises:
determining a second set of 3D points (par. 0040: “The set of virtual objects and set of anchors may be collectively sent to the second mobile computing device 221 as a set of augmented reality data 225, which may be associated with a different set of virtual model parameters than the first set of augmented reality data 215. The second mobile computing device 221 may then use the set of augmented reality data 225 to display the second set of virtual objects anchored to the second set of anchors stored in the augmented reality data 225. As shown by this example, in some embodiments, a same feature(s) may trigger different versions of an application executing on different mobile devices to display different virtual objects based on their different profile values.”); and
for each 3D point of the second set of 3D points:
computing, using the forward operation, a second set of weights based on the 3D point and the set of 3D anchor points (par. 0058: “ In some embodiments, some or all of the weights or coefficients of the model may be calculated during training with a training set, like a set of labeled images. Some embodiments may execute a gradient descent optimization to reduce an error rate (or increase fitness) and select appropriate weighting and biases of the model.”); and
computing, using the inverse operation, a 2D point mapped to the 3D point based on the second set of weights and the set of 2D anchor points (par. 0018: “Target objects may be two-dimensional (2D) or three-dimensional (3D) objects, like structured images encoding data, or structured volumes encoding data, such as a three-dimensional implementation of a bar-code or QR code. Some embodiments may determine spatial data associated with the target object, such as a set of points (e.g., in world coordinate system or an image coordinate system) representing points on the detected edge of the target physical object, a bounding box of the target physical object, or a contour of the target physical object. Some embodiments may analyze visual or other data to determine a partial or entire 2D or 3D reconstruction of the target physical object, such as point-cloud or polygon mesh in the world coordinate system (i.e. “world-space coordinate system”) representing cloth topology, surface roughness, or the like.”).
Regarding claim 8, Busey and Elahie teach the method of claim 7. Busey further teaches wherein determining the second set of 3D points comprises:
for each mapped 2D point, comparing the mapped 2D point to a predefined boundary condition (par. 0029: “In some embodiments, the mobile computing device 116 may send a query to a database of the networked computing resource 118 to determine how many other items similar to the wearable item 130 are present within a known geographic range represented by a geographic boundary 150.”); and
responsive to the mapped 2D point satisfying the predefined boundary condition, determining the second set of 3D points based on the 3D point to which the mapped 2D point was mapped using a point selection technique (par. 0030: “After detecting the wearable item 140 within the geographic boundary 150, where the wearable item 140 may be a second version of the wearable item 130, some embodiments may then obtain and generate a second version of a virtual object first generated for the wearable item 130. In some embodiments, the second version of a virtual object may be displayed as being attached to the wearable item 140. Additionally, some embodiments may limit virtual object generation for wearable items outside of this geographic range, such as a wearable item 190 that is outside of the geographic boundary 150.”).
Regarding claim 9, Busey and Elahie teach the method of claim 8. Busey further teaches wherein the point selection technique comprises:
for each 3D point to which a 2D point was mapped, determining a set of neighboring 3D points using the 3D mesh (par. 0018: “Some embodiments may determine spatial data associated with the target object, such as a set of points (e.g., in world coordinate system or an image coordinate system) representing points on the detected edge of the target physical object, a bounding box of the target physical object, or a contour of the target physical object. Some embodiments may analyze visual or other data to determine a partial or entire 2D or 3D reconstruction of the target physical object, such as point-cloud or polygon mesh in the world coordinate system (i.e. “world-space coordinate system”) representing cloth topology, surface roughness, or the like. Some embodiments may infer a virtual representation target physical object (e.g., the point-cloud or polygon mesh in the world coordinate system), and in some cases, the virtual representation may include the contour (or other texture data) and the set of points.”);
for each 3D point of the set of neighboring 3D points:
determining whether the 3D point has previously been selected (par. 0075: “Some embodiments may transmit a set of messages indicating a user's selections of new virtual objects. These messages may be transmitted to a remote server and accessed by other mobile devices or other applications. For example, a user may select a first virtual object and a second virtual object, and, in response, some embodiments may send a message indicating the selection of the first virtual object and second virtual object to a remote server to be stored in a record of the database (e.g. a user profile, a scoreboard, and the like). The record storing the selection of the first virtual object and second virtual object may then be accessed by a second instance of the application operating on a second mobile device.”); and
responsive to determining that the 3D point has not been previously selected, add the 3D point to the second set of 3D points (par. 0075, as above).
Regarding claim 10, Busey and Elahie teach the method of claim 8. Busey further teaches wherein the point selection technique comprises:
determining one or more partitions of the 3D mesh, each partition comprising one or more 3D points of the plurality of 3D points (par. 0018: “Some embodiments may analyze visual or other data to determine a partial or entire 2D or 3D reconstruction of the target physical object, such as point-cloud or polygon mesh in the world coordinate system (i.e. “world-space coordinate system”) representing cloth topology, surface roughness, or the like.”);
for each 3D point to which a 2D point was mapped, determining a set of neighboring 3D points using the 3D mesh (par. 0021: “Images can be 2D images or 3D images, having a depth channel with distance of pixels from the headset, for instance inferred with depth sensors or visual simultaneous localization and mapping algorithms (like direct sparse odometry, semi-direct visual odometry, LSD-SLAM, DTAM, RGB-D vSLAM, Direct RGB-D Odometry, or others), which in some cases may be executed on-device 116, in real-time (like updating a map within less than 20, 50, or 200 milliseconds of obtaining an image).”);
for each 3D point of the set of neighboring 3D points:
determining a partition that includes the 3D point (par. 0018, as above);
for each 3D point of the partition that includes the 3D point:
determining whether the 3D point has previously been selected (par. 0075, as above in claim 9 rejection); and
responsive to determining that the 3D point has not been previously selected, add the 3D point to the second set of 3D points (par. 0075, as above in claim 9 rejection).
Regarding claim 11, Busey and Elahie teach the method of claim 8. Busey further teaches wherein the predefined boundary condition is based on inclusion in a convex hull defined by the set of 2D anchor points (par. 0029: “In some embodiments, the mobile computing device 116 may send a query to a database of the networked computing resource 118 to determine how many other items similar to the wearable item 130 are present within a known geographic range represented by a geographic boundary 150.”).
Regarding claim 12, Busey and Elahie teach the method of claim 7. Busey further teaches wherein each 3D point of the first set of 3D points and each 3D point of the second set of 3D points are processed in parallel (par. 0117: “Multiple processors may be employed to provide for parallel or sequential execution of one or more portions of the techniques described herein.”).
Claim 13 is substantially similar to claim 1, except that it teaches a system rather than a method. As such, it is rejected on a similar basis to claim 1.
Claim 14 is substantially similar to claim 9, except that it depends from claim 13 rather than claim 8. As such, it is rejected on a similar basis to claim 9.
Claim 15 is substantially similar to claim 10, except that it depends from claim 13 rather than claim 8. As such, it is rejected on a similar basis to claim 10.
Claim 16 is substantially similar to claim 7, except that it depends from claim 13 rather than claim 5. As such, it is rejected on a similar basis to claim 7.
Claim 17 is substantially similar to claim 1, except that it teaches a non-transitory computer-readable medium rather than a method. As such, it is rejected on a similar basis to claim 1.
Claim 18 is substantially similar to claim 9, except that it depends from claim 17 rather than claim 7. As such, it is rejected on a similar basis to claim 9.
Claim 19 is substantially similar to claim 10, except that it depends from claim 17 rather than claim 8. As such, it is rejected on a similar basis to claim 10.
Claim 20 is substantially similar to claim 4, except that it depends from claim 17 rather than claim 3. As such, it is rejected on a similar basis to claim 4.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN A BARHAM whose telephone number is (571)272-4338. The examiner can normally be reached Mon-Fri, 8:30am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu, can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN ALLEN BARHAM/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights