DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: a computer implemented pose estimator configured for, and a computer implemented rendering unit in claim 14.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 12, 14 – 19, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Swaminathan et al. (Publication: US 2020/0410766 A1) in view of Oh et al. (Publication: US 2019/0293609 A1).
Regarding claim 1, Swaminathan discloses a computer implemented method comprising (Fig. 37, [0283] As illustrated, gateway 3740 is implemented as a separate hardware component, which includes a processor for executing software instructions and memory for storing software instructions and data. [0121] - methods and apparatus for caching and updating 3D reconstruction data.):
receiving a 3D textured model of a part that includes inspection collected for an industrial process associated with producing the part (
[0307], Fig. 39 - 3902, a device may capture 3D information about a physical world including objects, references, in the physical world.
[0121] - methods and apparatus for caching and updating 3D reconstruction data. [0049] FIG. 18 shows a table that is used by a method of categorizing all the pixels in the rectangular with respect to a minimum brick value (bmin) and a maximum brick value (bmax) in FIG. 17.
[0134] - the fixed element may be a table and the virtual content may be positioned such that it appears to be on that table. the AR content may be placed within structures in a field of view 44, which may be a present field of view or an estimated future field of view. the AR content may be placed relative to a mapped mesh model 46 of the physical world.);
tracking a viewing perspective of a user relative to a reference part ([0161] A world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved. As the sensor's frame pose is known and how it relates to the other poses, each of these multiple viewpoints of the physical world may be fused together into a single, combined reconstruction.
[0134] - the fixed element may be a table and the virtual content may be positioned such that it appears to be on that table. the AR content may be placed within structures in a field of view 44, which may be a present field of view or an estimated future field of view. the AR content may be placed relative to a mapped mesh model 46 of the physical world.);
generating pose data based on the tracked viewing perspective ([0161] A world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved. As the sensor's frame pose is known and how it relates to the other poses, each of these multiple viewpoints of the physical world may be fused together into a single, combined reconstruction.
[0311], Fig. 39 – where the processor may identify a subset of blocks corresponding to a portion of the physical world required to deliver 3D reconstruction data in accordance with the request. The identification of blocks may be based on, for example, data collected by a sensor (e.g., depth sensor 51, world camera 52, inertial measurement units 57, global positioning system, and/or the like). A multi-device system may create a common coordinate frame such that blocks generated by different devices associated with corresponding portions of the physical world may be created using the common coordinate frame without regard to which device provided the 3D reconstruction data to reconstruct the portion of the physical world represented by that block. As one example of how a common coordinate frame may be created, data from devices in the same general vicinity may be routed to the same server or one or more servers for processing. There, data from each device may be initially represented in a device-specific coordinate frame. Once sufficient data from each of the devices has been gathered to identify features in a common portion of the physical world, those features may be correlated, providing the transformation from one device-specific coordinate frame to the others. One of these device-specific coordinate frames may be designated as the common coordinate frame and the transformations between the other coordinate frames, and that coordinate frame may be used to convert data from the device-specific coordinate frames to the coordinate frame designated as the common coordinate frame.);
rendering an image that includes inspection from the 3D textured model based on the pose data ([0161] A world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved. As the sensor's frame pose is known and how it relates to the other poses, each of these multiple viewpoints of the physical world may be fused together into a single, combined reconstruction. . [0316] At act 4004, a processor (e.g., processor 3812 or 3822) of the system may create versions of blocks including 3D reconstruction data of the physical world based on the 3D information captured by the one or more sensors. each block may be formatted as one or more portions of a mesh.
[0317] The blocks may have versions, such that each time information about a region of the physical world is captured by any device, a new version of the block may be stored. Each version of the block may have 3D reconstruction data including values representing objects in a region of the physical world at a point in time. such processing may be performed locally on the device, resulting in new versions of blocks being stored in active memory. in a multi-device system, similar processing may be performed in a server (e.g., server 3710 of FIG. 37), which may manage versions of the blocks such that the most recent version of each block available in its remote cache is supplied when requested by any device.).
Swaminathan does not Oh discloses
inspection data ([0038] Discerning materials and/or textures of real-world objects from captured video and/or images, for example, using a lightweight and/or portable device such as a smartphone, may be useful to detect materials and/or textures of the objects without touching the objects by hand. For example, an object may be corrosive or dangerous to handle by hand, and identifying the material of the object using a smartphone (or HMD) without touching the object may be useful in such a scenario to avoid injury.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Swaminathan with inspection data as taught by Oh. The motivation for doing is to have less costly method.
Regarding claim 2, see rejection on claim 15.
Regarding claim 3, see rejection on claim 16.
Regarding claim 4, see rejection on claim 17.
Regarding claim 5, see rejection on claim 18.
Regarding claim 6, see rejection on claim 19.
Regarding claim 7, see rejection on claim 20.
Regarding claim 8, Swaminathan in view Ho disclose all the limitation of claim 1 including prediction data.
Swaminathan discloses receiving a input selecting a region of the reference part, the rendering comprising displaying an indication of the input ([0134] - the AR content may be placed by appropriately selecting portions of a fixed element 42 (e.g., a table) from a reconstruction (e.g., the reconstruction 318) to determine the shape and position of the AR content 40. As an example, the fixed element may be a table and the virtual content may be positioned such that it appears to be on that table. In the AR content may be placed within structures in a field of view 44, which may be a present field of view or an estimated future field of view. the AR content may be placed relative to a mapped mesh model 46 of the physical world. ).
Oh discloses receiving a user input, user input ([0056] - the target object for material/texture identification can be specified (e.g., selected by the user)).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Swaminathan in view of Oh with inspection data as taught by Oh. The motivation for doing is to offer different ways of inputting for convenience.
Regarding claim 9, Swaminathan in view Ho disclose all the limitation of claim 1 including prediction data.
Swaminathan discloses includes data collected from multiple industrial processes associated with producing the part ([0193] FIG. 17, the following process, image enhancer, shows an exemplary method 908 of conducting a second depth image acceptance test, . The method 908 may start at act 1102 by categorizing all the pixels in the rectangle with respect to the bmin value and the bmax value. At act 1104, the method 908 may determine whether the tested brick is in front of a solid or holey background, for example, by using a table shown in FIG. 18. If it is determined that the tested brick is in front of a solid or holey background, the method 908 may accept (act 1106) the brick. If it is determined that the tested brick is not in front of a solid or holey background, at act 916 the method 908 may cull the brick “multiple industrial process”.
[0095] environmental reasoning may involve identifying clear surfaces by recognizing that they are window panes or glass table tops. From such an identification, regions that contain physical objects might be classified as not occluding virtual objects but might be classified as interacting with virtual objects.),
the method comprising selectively including, in the rendered image, the data collected from different industrial processes, based on detected inputs ([0268] In some embodiments, 3D reconstruction data may be segmented into blocks. The 3D reconstruction data may be transmitted among storage mediums on the basis of blocks.
[0134] - the AR content may be placed by appropriately selecting portions of a fixed element 42 (e.g., a table) from a reconstruction (e.g., the reconstruction 318) to determine the shape and position of the AR content 40. As an example, the fixed element may be a table and the virtual content may be positioned such that it appears to be on that table. In the AR content may be placed within structures in a field of view 44, which may be a present field of view or an estimated future field of view. the AR content may be placed relative to a mapped mesh model 46 of the physical world.).
Oh discloses selectively based on detected user inputs ([0056] FIG. 5 is a flowchart of an exemplary method for identifying material and/or texture of an object by employing sound. In step 502 of the exemplary method 500, a target object is selected. Some HMDs (e.g., smart glasses for AR) are able to recognize a user's hand gesture and/or are able to track movement of a user's pointing finger and/or of a user's eyes. By using such technology, the target object for material/texture identification can be specified (e.g., selected by the user). In some embodiments, additionally, or alternatively, a focus mark is displayed on a glass display portion of the HMD and can be employed for object selection. FIG. 6 illustrates an example overview of target-object selection. The example pair of smart glasses 602 is configured to recognize/track a movement and/or a direction of a user's hand gesture, pointing finger, and/or eyes, and is configured to determine object-selection data based on the recognized/tracked user movement/direction. As illustrated in FIG. 6, the smart glasses 602 may determine target-object selection data that may be indicative of movement of a user's finger with respect to the target object. The smart glasses 602 may determine target-object selection data that may be indicative of movement of a user's eyes with respect to the target object. As further illustrated in FIG. 6, the smart glasses 602 may determine target-object selection data based on a position of a focus mark that is displayed to the user. In either scenario, the target-object selection data may be indicative of a direction to emit a sound signal so that the sound signal may reflect off the target object. The target object may be selected so that a directional sound signal can be emitted in a direction of the selected target object. In some embodiments, an adjacent object located near the selected target object can be a secondary target object.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Swaminathan with selectively based on detected user inputs as taught by Oh. The motivation for doing is to offer different ways of inputting for convenience.
Regarding claim 10, Swaminathan in view Ho disclose all the limitation of claim 9 including prediction data.
Swaminathan discloses
the tracking and the rendering uses one or more of an augmented/mixed reality headset including a head-mounted digital display and camera ([0364] - a head-mounted display device comprising a depth camera and/or visual cameras. [0161] A world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved.);
the tracking and the rendering uses a mobile device including a digital display and connected camera;
the tracking and the rendering uses a separate camera and digital display; or
the tracking and the rendering uses a camera and digital projector; and the rendering comprises overlaying computer-generated data derived from the inspection over physical world images of the reference part.
Regarding claim 11, Swaminathan in view Ho disclose all the limitation of claim 1 including prediction data.
Swaminathan discloses wherein the reference part is the same physical part as the part ([0161] A world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved. As the sensor's frame pose is known and how it relates to the other poses, each of these multiple viewpoints of the physical world may be fused together into a single, combined reconstruction.
[0134] - the fixed element may be a table and the virtual content may be positioned such that it appears to be on that table. the AR content may be placed within structures in a field of view 44, which may be a present field of view or an estimated future field of view. the AR content may be placed relative to a mapped mesh model 46 of the physical world, physical part.).
Regarding claim 12, Swaminathan in view Ho disclose all the limitation of claim 1 including prediction data.
Swaminathan discloses wherein the reference part is a proxy object for the part ([0134] - the fixed element may be a table and the virtual content may be positioned such that it appears to be on that table. the AR content may be placed within structures in a field of view 44, which may be a present field of view or an estimated future field of view. the AR content may be placed relative to a mapped mesh model 46 of the physical world “proxy object”.).
Regarding claim 14, Swaminathan discloses an interactive inspection system comprising (Fig. 37, [0283] As illustrated, gateway 3740 is implemented as a separate hardware component, which includes a processor for executing software instructions and memory for storing software instructions and data. [0121] - methods and apparatus for caching and updating 3D reconstruction data.):
data storage storing a 3D textured model of a part that includes inspection collected for an industrial process associated with producing the part (Fig. 37, [0283] As illustrated, gateway 3740 is implemented as a separate hardware component, which includes a processor for executing software instructions and memory for storing software instructions and data. [0121] - methods and apparatus for caching and updating 3D reconstruction data.
[0307], Fig. 39 - 3902, a device may capture 3D information, stored in data storage, about a physical world including objects, references, in the physical world. );
a tracking device configured for tracking a viewing perspective of a user relative to a reference part (0161] A world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved. As the sensor's frame pose is known and how it relates to the other poses, each of these multiple viewpoints of the physical world may be fused together into a single, combined reconstruction.
[0134] - the fixed element may be a table and the virtual content may be positioned such that it appears to be on that table. the AR content may be placed within structures in a field of view 44, which may be a present field of view or an estimated future field of view. the AR content may be placed relative to a mapped mesh model 46 of the physical world.);
a computer implemented pose estimator configured for generating pose data indicating a pose of the reference part in reliance on the tracked viewing perspective ([0161] A world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved. As the sensor's frame pose is known and how it relates to the other poses, each of these multiple viewpoints of the physical world may be fused together into a single, combined reconstruction.
[0311], Fig. 39 – where the processor may identify a subset of blocks corresponding to a portion of the physical world required to deliver 3D reconstruction data in accordance with the request. The identification of blocks may be based on, for example, data collected by a sensor (e.g., depth sensor 51, world camera 52, inertial measurement units 57, global positioning system, and/or the like). A multi-device system may create a common coordinate frame such that blocks generated by different devices associated with corresponding portions of the physical world may be created using the common coordinate frame without regard to which device provided the 3D reconstruction data to reconstruct the portion of the physical world represented by that block. As one example of how a common coordinate frame may be created, data from devices in the same general vicinity may be routed to the same server or one or more servers for processing. There, data from each device may be initially represented in a device-specific coordinate frame. Once sufficient data from each of the devices has been gathered to identify features in a common portion of the physical world, those features may be correlated, providing the transformation from one device-specific coordinate frame to the others. One of these device-specific coordinate frames may be designated as the common coordinate frame and the transformations between the other coordinate frames, and that coordinate frame may be used to convert data from the device-specific coordinate frames to the coordinate frame designated as the common coordinate frame.);
a computer implemented rendering unit configured for generating a rendered image of the part based on the pose data, the rendered image including inspection from the 3D textured model ([0161] A world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved. As the sensor's frame pose is known and how it relates to the other poses, each of these multiple viewpoints of the physical world may be fused together into a single, combined reconstruction. . [0316] At act 4004, a processor (e.g., processor 3812 or 3822) of the system may create versions of blocks including 3D reconstruction data of the physical world based on the 3D information captured by the one or more sensors. each block may be formatted as one or more portions of a mesh.
[0317] The blocks may have versions, such that each time information about a region of the physical world is captured by any device, a new version of the block may be stored. Each version of the block may have 3D reconstruction data including values representing objects in a region of the physical world at a point in time. such processing may be performed locally on the device, resulting in new versions of blocks being stored in active memory. in a multi-device system, similar processing may be performed in a server (e.g., server 3710 of FIG. 37), which may manage versions of the blocks such that the most recent version of each block available in its remote cache is supplied when requested by any device.); and
a display device configured for displaying the rendered image for the user ([0268] 3D reconstruction data may be segmented into blocks. The 3D reconstruction data may be transmitted among storage mediums on the basis of blocks. For example, a block may be paged out of an active memory and persisted to a local or remote cache. The system may implement a paging algorithm in which active memory associated with a wearable device (e.g., head-mounted display device) stores blocks representative of a portion of a 3D reconstruction of the physical world in a field of view of a user of the wearable device.).
Swaminathan does not Oh discloses
inspection data ([0038] Discerning materials and/or textures of real-world objects from captured video and/or images, for example, using a lightweight and/or portable device such as a smartphone, may be useful to detect materials and/or textures of the objects without touching the objects by hand. For example, an object may be corrosive or dangerous to handle by hand, and identifying the material of the object using a smartphone (or HMD) without touching the object may be useful in such a scenario to avoid injury.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Swaminathan with inspection data as taught by Oh. The motivation for doing is to have less costly method.
Regarding claim 15, Swaminathan in view Ho disclose all the limitation of claim 14 including prediction data.
Swaminathan discloses the tracking device includes a camera receiving video image data that includes successive images of the reference part ([0307], Fig. 39 - 3902, a device may capture 3D information about a physical world including objects, references, in the physical world. );
the pose estimator generates the pose data by mapping, based on the video image data, physical locations and orientations of the reference part over the successive images to a reference coordinate system ([0161] A world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved. As the sensor's frame pose is known and how it relates to the other poses, each of these multiple viewpoints of the physical world may be fused together into a single, combined reconstruction.
[0311], Fig. 39 – following instructions, pose estimator: 3906, where the processor may identify a subset of blocks corresponding to a portion of the physical world required to deliver 3D reconstruction data in accordance with the request. The identification of blocks may be based on, for example, data collected by a sensor (e.g., depth sensor 51, world camera 52, inertial measurement units 57, global positioning system, and/or the like). A multi-device system may create a common coordinate frame such that blocks generated by different devices associated with corresponding portions of the physical world may be created using the common coordinate frame without regard to which device provided the 3D reconstruction data to reconstruct the portion of the physical world represented by that block. As one example of how a common coordinate frame may be created, data from devices in the same general vicinity may be routed to the same server or one or more servers for processing. There, data from each device may be initially represented in a device-specific coordinate frame. Once sufficient data from each of the devices has been gathered to identify features in a common portion of the physical world, those features may be correlated, providing the transformation from one device-specific coordinate frame to the others. One of these device-specific coordinate frames may be designated as the common coordinate frame and the transformations between the other coordinate frames, and that coordinate frame may be used to convert data from the device-specific coordinate frames to the coordinate frame designated as the common coordinate frame.); and the
rendering unit maps inspection from the 3D textured model to the reference part over the successive images based on the pose data and generates successive corresponding rendered images including the mapped inspection ([0161] A world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved. As the sensor's frame pose is known and how it relates to the other poses, each of these multiple viewpoints of the physical world may be fused together into a single, combined reconstruction. . [0316] At act 4004, a processor (e.g., processor 3812 or 3822) of the system may create versions of blocks including 3D reconstruction data of the physical world based on the 3D information captured by the one or more sensors. each block may be formatted as one or more portions of a mesh.
[0317] The blocks may have versions, such that each time information about a region of the physical world is captured by any device, a new version of the block may be stored. Each version of the block may have 3D reconstruction data including values representing objects in a region of the physical world at a point in time. such processing may be performed locally on the device, resulting in new versions of blocks being stored in active memory. in a multi-device system, similar processing may be performed in a server (e.g., server 3710 of FIG. 37), which may manage versions of the blocks such that the most recent version of each block available in its remote cache is supplied when requested by any device.).
Regarding claim 16, Swaminathan in view Ho disclose all the limitation of claim 15 including prediction data.
Swaminathan discloses wherein the reference part is a physical part having an actual geometry that varies relative to geometry data of the 3D textured model of the part ([0370] Referring back to FIG. 49, The raycast engine 4906 may generate a view of the physical world given a user's pose, and may remove the holes out of the depth map. The data may represent portions of a user's current view of the physical world at a current time, “reference part”.
[0134] - the AR content may be placed by appropriately selecting portions of a fixed element 42 (e.g., a table) from a reconstruction (e.g., the reconstruction 318) to determine the shape and position of the AR content 40. As an example, the fixed element may be a table and the virtual content may be positioned such that it appears to be on that table. the AR content may be placed within structures in a field of view 44, which may be a present field of view or an estimated future field of view. the AR content may be placed relative to a mapped mesh model 46 of the physical world.
[0277] - sensor may be a depth camera, which may capture 3D information of the environment, for example, a stream of depth images with respective poses of the depth camera (i.e. camera poses). The 3D information of the environment may be processed into a voxel grid. Each voxel may contain one or more signed distance functions (SDFs) that describe whether the voxel lies inside or outside the geometries of objects in the environment.), and
the system further includes a geometry variation module configured to compute geometry variation data that enables the geometry data for the 3D textured model to be conformed to the actual geometry of the reference part, wherein the rendered images are generated by the rendering unit based on both the pose data and the geometry variation data ([0370] Referring back to FIG. 49, The raycast engine 4906 may generate a view of the physical world given a user's pose, and may remove the holes out of the depth map. The data may represent portions of a user's current view of the physical world at a current time, “reference part”.
[0134] - the AR content may be placed by appropriately selecting portions of a fixed element 42 (e.g., a table) from a reconstruction (e.g., the reconstruction 318) to determine the shape and position of the AR content 40. As an example, the fixed element may be a table and the virtual content may be positioned such that it appears to be on that table. the AR content may be placed within structures in a field of view 44, which may be a present field of view or an estimated future field of view. the AR content may be placed relative to a mapped mesh model 46 of the physical world.
[0277] - the following process, geometry variation module, sensor may be a depth camera, which may capture 3D information of the environment, for example, a stream of depth images with respective poses of the depth camera (i.e. camera poses). The 3D information of the environment may be processed into a voxel grid. Each voxel may contain one or more signed distance functions (SDFs) that describe whether the voxel lies inside or outside the geometries of objects in the environment.).
Regarding claim 17, Swaminathan in view Ho disclose all the limitation of claim 16 including prediction data.
Swaminathan discloses using a process that includes perturbing 3D model geometry data to determine offsets for a plurality of reference points of the 3D model geometry data to corresponding points of the reference part ([0049], [0182] - FIG. 18 shows a table that is used by a method of categorizing all the pixels in the rectangular with respect to a minimum brick value (bmin) and a maximum brick value (bmax) in FIG. 17. 600 can categorize bricks by their distance to a surface with respect to a truncated threshold. For example, the method 600 can identify empty bricks (e.g., the bricks that are culled, or the bricks that are away from the surface beyond the truncated threshold) so as to not allocate memory space for the empty bricks. The method 600 can also identify bricks that are away from the surface by the truncated threshold so as to store these bricks by a constant distance value of a negative truncation threshold and weight 1. The method 600 can also identify bricks that have a distance to the surface between zero and the truncated threshold so as to store these bricks with a constant SDF value of the positive truncation threshold, but varying weight. Storing the distance or weight values, which are constant for a brick with a single value, may be an entropy-based compression for a zero-entropy field “determine offset”, surface, FOV is the reference part.
Fig. 12, step 610: generate data for apportion of the 3D reconstruction of the surface in the scene based on the bricks, 3D model.
PNG
media_image1.png
464
558
media_image1.png
Greyscale
PNG
media_image2.png
538
806
media_image2.png
Greyscale
).
Regarding claim 18, Swaminathan in view Ho disclose all the limitation of claim 15 including prediction data.
Swaminathan discloses data corresponding to one or more of: near-infrared (NIR) image data, infrared (IR) image data, and/or visible light image data ( [0113] - infrared camera sensors. [0375] - the depth sensor uses infrared (IR) light, the holes may result, for example, from materials or structures in the physical environment).
Ho discloses wherein the rendered images are augmented with the texture data overlaid on the images of the part ([0037] Recognizing a material and/or a texture of a target object may be useful to overlay augmented information on the target object realistically. For example, identifying a material and/or a texture of one or more real objects from captured video and/or images (e.g., captured by a head-mounted display (HMD)) may enable improved tracking of the one or more real objects and/or may enable more realistic/accurate overlay of augmented information on the one or more real objects. Interactions of AR objects with real-world objects may be based on material and/or texture determinations of the real-world objects. For example, if a floor/ground of a real-world environment is determined to be hardwood, an AR character may be augmented to walk or slide on the hardwood floor differently than if the floor/ground of the real-world environment is determined to be sand. As another example, if a portion of a wall of a real-world environment is determined to be glass, an AR character shining a flashlight may be augmented differently (e.g., taking into account interactions of a shining flashlight with glass) than if the portion of the wall is determined to be brick.) .
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Swaminathan in view of Ho with wherein the rendered images are augmented with the texture data overlaid on the images of the part as taught by Oh. The motivation for doing is to have less costly method.
Regarding claim 19, Swaminathan in view Ho disclose all the limitation of claim 15 including prediction data.
Swaminathan discloses an image enhancer associated with the rendering unit, the image enhancer configured to process the inspection collected for the industrial process associated with producing the part to visually enhance features of interest in the rendered images ([0193] FIG. 17, the following process , image enhancer, shows an exemplary method 908 of conducting a second depth image acceptance test, . The method 908 may start at act 1102 by categorizing all the pixels in the rectangle with respect to the bmin value and the bmax value. At act 1104, the method 908 may determine whether the tested brick is in front of a solid or holey background, for example, by using a table shown in FIG. 18. If it is determined that the tested brick is in front of a solid or holey background, the method 908 may accept (act 1106) the brick. If it is determined that the tested brick is not in front of a solid or holey background, at act 916 the method 908 may cull the brick “classification”. [0194] - if no culling is performed, a world reconstruction component would compute all of the 256 bricks. With the brick culling against the camera frustum, a world reconstruction component only needs to compute the first plurality of bricks, i.e. 34 bricks, and thus would render a result much faster, after image enhancer process then performs the rendering “image enhancer associated with the rendering unit”.
[0095] environmental reasoning may involve identifying clear surfaces by recognizing that they are window panes or glass table tops. From such an identification, regions that contain physical objects might be classified as not occluding virtual objects but might be classified as interacting with virtual objects.).
.
Regarding claim 21, Swaminathan in view Ho disclose all the limitation of claim 14 including prediction data.
Swaminathan discloses
the tracking device and the display device are implemented using an augmented reality headset that includes a head-mounted digital display and camera ([0364] - a head-mounted display device comprising a depth camera and/or visual cameras. [0161] A world reconstruction system may integrate sensor data over time from multiple viewpoints of a physical world. The poses of the sensors (e.g., position and orientation) may be tracked as a device including the sensors is moved.);
the tracking device and the display device are implemented using mobile device that includes a digital display and connected camera;
the tracking device and the display device comprise a separate camera and digital display, respectively; or
the tracking device and the display device comprise a camera and digital projector, respectively; and the rendered image comprises a computer-generated data derived from the inspection overlaid onto a physical world images of the reference part.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Swaminathan et al. (Publication: US 2020/0410766 A1) in view of Oh et al. (Publication: US 2019/0293609 A1) and Marzorati et al. (Publication: US 2021/0225052 A1).
Regarding claim 20, Swaminathan in view Ho disclose all the limitation of claim 19 including prediction data.
Swaminathan discloses includes a function for generating classification data for one or more regions of the part based on the inspection, the classification data being on a representation of the part in the rendered images ([0193] FIG. 17, the following process, image enhancer, shows an exemplary method 908 of conducting a second depth image acceptance test, . The method 908 may start at act 1102 by categorizing all the pixels in the rectangle with respect to the bmin value and the bmax value. At act 1104, the method 908 may determine whether the tested brick is in front of a solid or holey background, for example, by using a table shown in FIG. 18. If it is determined that the tested brick is in front of a solid or holey background, the method 908 may accept (act 1106) the brick. If it is determined that the tested brick is not in front of a solid or holey background, at act 916 the method 908 may cull the brick “classification”.
[0095] environmental reasoning may involve identifying clear surfaces by recognizing that they are window panes or glass table tops. From such an identification, regions that contain physical objects might be classified as not occluding virtual objects but might be classified as interacting with virtual objects.).
Swaminathan in view of Ho do not, Marzorati discloses a machine learned prediction function for generating data, the data being overlaid on a representation of the part in the rendered images ([0002] - As new images are received and analyzed to identify products (e.g., food items) and related product contents (e.g., food ingredients), the artificial intelligence machine learning techniques are applied to the new image to predict a new image relevancy which may be evaluated against a predetermined relevancy threshold (e.g., a calorie count or limit) when determining whether to overlay or augment the new image with
real-time information related to the predicted new image relevancy.
[0019] - The personalized information overlay device may detect objects displayed in received video/images, retrieve associated ingredients or contents of each identified object from a health corpus of medical or dietary information, model the health-related interactions between the consumer and the associated ingredients or contents using a machine learning model and user-specified health criteria, and present personalized health-related guidance output based on the model by displaying personalized health-related information which augments and/or overlays the displayed video/image.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Swaminathan in view of Ho with a machine learned prediction function for generating data, the data being overlaid on a representation of the part in the rendered images as taught by Marzorati. The motivation for doing is to provide suitable product.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ming Wu whose telephone number is (571)270-0724. The examiner can normally be reached on Monday - Friday: 9:30am - 6:00pm EST .
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MING WU/
Primary Examiner, Art Unit 2618