Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities: page 9, paragraph 1 alternately refers to probability function 212 as either “function_1” or “funciton_1.” This function should be labeled consistently throughout the specification.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 5-9 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Bradski (US 20190094981 A1).
Regarding claim 5, Bradski teaches a method for using a sensor system to update at least one virtual object based on a data structure (par. 0783: “Specifically, data received from the systems' cameras 5120 and data received from sensors such as IMUs 5122 may be utilized to determine a pose at which various images were captured. This information allows the system to place one or more map points derived from the images at the appropriate position and orientation in the Map 5106.”),
wherein the data structure is representative of the at least one virtual object[[s]] (par. 0782: “In one or more embodiments, the one or more objects may be recognized previously and stored in the map database. In other embodiments, if the information is new, object recognizers may run on the new data, and the data may be transmitted to one or more wearable AR systems (5008). Based on the recognized real objects and/or other information conveyed to the AR system, the desired virtual scene may be accordingly displayed to the user of the wearable AR system (5010).”),
wherein the data structure comprises a first portion comprising constant data and a second portion comprising variable data (par. 0990: “Further, the user interface components may employ various types of environmental data, for example GPS location data, Wi-Fi signal strength date, cellphone differential signal strength, known features, image histogram profiles, hashes of room features, etc., proximity to walls/ceiling/floors/3D-blobs/etc., location in the world (e.g., home, office, car, street), approximate social data (e.g., “friends”), and/or voice recognition.”)?,
the method comprising:
controlling the sensor system to search for a real object[[s]] (par. 0785: “As shown in FIG. 51, data from the Map 5106 is transmitted as needed to provide an AR experience to a plurality of users of the wearable AR system. One or more users may interact with the AR system through gesture tracking 5128, eye tracking 5130, totem tracking 5132 or through a gaming console 5134.”),
wherein the real object[[s]] correspond to the at least one virtual object[[s]] (par. 0786: “In one or more embodiments, the Map 5106 may comprise a set of raster imagery, point+descriptors clouds and/or polygonal/geometric definitions corresponding to one or more objects of the real world.”),
wherein the at least one virtual object[[s]] is represented in the variable data based on a range of values and/or probability functions of at least one variable element[[(s)]] (par. 0641: “One approach to find new points that avoids such a large search operation is by render rather than search. In other words, assuming the position of M keyframes are known and each of them has N points, the AR system may project lines (or cones) from N features to the M keyframes to triangulate a 3D position of the various 2D points. Referring now to FIG. 37, in this particular example, there are 6 keyframes 3702, and lines or rays are rendered (using a graphics card) from the 6 keyframes to the points 3704 derived from the respective keyframe. In one or more embodiments, new 3D map points may be determined based on the intersection of the rendered lines. In other words, when two rendered lines intersect, the pixel coordinates of that particular map point in a 3D space may be 2 instead of 1 or 0. Thus, the higher the intersection of the lines at a particular point, the higher the likelihood is that there is a map point corresponding to a particular feature in the 3D space. In one or more embodiments, this intersection approach, as shown in FIG. 37 may be used to find new map points in a 3D space.”);
obtaining sensor data from the sensor system (par. 0774: “On a basic level, the AR system 4900 may receive input (e.g., visual input 4902 from the user's wearable system, input from room cameras, sensory input in the form of various sensors in the system, gestures, totems, eye tracking etc.) from one or more AR systems.”),
wherein the sensor data is representative of physical properties of the real object[[s]] (par. 0774: “The wearable AR systems not only provide images from the cameras, they may also be equipped with various sensors (e.g., accelerometers, temperature sensors, movement sensors, depth sensors, GPS, etc.) to determine the location, and various other attributes of the environment of the user.”),
updating at least some of the variable data of the at least one virtual object based on the sensor data (par. 0787: “The Map 5106 is constantly updated with information received from multiple augmented reality devices, and becomes more and more accurate over time.”); and
displaying the updated at least one virtual object (par. 0787: “Also, the processor/controller may determine through the various components (e.g., fusion process, pose process, stereo, etc.) a set of output parameters that can be used to project a set of images to the user through a suitable vision system.”).
Regarding claim 6, Bradski teaches the method of claim 5, wherein displaying the updated at least one virtual object is based on some or all of the constant data (par. 0782: “Based on the recognized real objects and/or other information conveyed to the AR system, the desired virtual scene may be accordingly displayed to the user of the wearable AR system (5010).”).
Regarding claim 7, Bradski teaches the method of claim 5, wherein the sensor system is arranged to update the sensor data (par. 0791: “Asynchronous communications is established between the user's respective individual AR system and the cloud based computers (e.g., server computers). In other words, the user's individual AR system is constantly updating information about the user's surroundings to the cloud, and also receiving information from the cloud about the passable world.”),
wherein the displaying is repeated each time the sensor data is updated (par. 0820: “For example, the calibration may produce inputs that relate to where the cameras are relative to a helmet or other head-worn module; the global reference of the helmet; the intrinsic parameters of the cameras, etc. such that the system can adjust the images in real-time in order to determine a location of every pixel in an image in terms of ray direction in space.”).
Regarding claim 8, Bradski teaches the method of claim 5, wherein the range of values and/or the probability distribution of a first variable element of the variable data is based on at least one item selected from the group consisting of previous values of the first variable element, values of a second variable element, or previous values of the second variable element (par. 0756: “The approach described herein provides a very complex artificial intelligence (AI) property by performing deterministic acts with completely deterministic globally visible mechanisms for transitioning from one state to another. These actions are implicitly map-able to a behavior that a user cares about. Constant insight through monitoring of these global values of an overall state of the system is required, which allows the insertion of other states or changes to the current state.”).
Regarding claim 9, Bradski teaches the method of claim 5, wherein the sensor data comprises at least one of visual sensor data (par. 0200: “The user sensing system 34 may also include one or more infrared camera sensors, one or more visible spectrum camera sensors, structured light emitters and/or sensors, infrared light emitters, coherent light emitters and/or sensors, gyros, accelerometers, magnetometers, proximity sensors, GPS sensors, ultrasonic emitters and detectors and haptic interfaces.”), infrared sensor data (par. 0200, as above), microwave sensor data, ultrasound sensor data, audio sensor data (par. 0205: “in some embodiments, the environment-sensing system 36 may include a microphone for receiving audio from the local environment.”), position sensor data, accelerometer sensor data (par. 0200, as above), or global positioning system data (par. 0200, as above).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4 and 13-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Walczak (“X-VRML – XML Based Modeling of Virtual Reality”), and further in view of Detry (“A Probabilistic Framework for 3D Visual Object Representation”).
Regarding claim 1, Walczak teaches a method comprising:
Storing a data structure in a non-volatile computer storage (p. 2, par. 6: “These virtual scene models are coded by the use of a high-level language called X-VRML. The language has higher expression power than VRML, allows customization of the virtual world generation process, enables selection of the virtual world contents, and allows access to data stored in database systems, or files.”),
wherein the data structure comprises a first portion comprising constant data and a second portion comprising variable data (p. 4, col. 1: “In X-VRML, all values of element attributes are treated as expressions and are evaluated prior to element interpretation. Expressions can contain: • constant numerical, textual, and Boolean values; • variable references; • operators; and • functions.”),
wherein the constant data corresponds to constant physical properties of at least one virtual object[[s]] (p. 3, par. 8: “The X-VRML language is an application of the Extensible Markup Language (XML) [5] and defines a set of XML tags that can be used in X-VRML models. According to XML syntax, tags have names and parameters. A parameter has a name and a text value. In X-VRML, the value may be a constant or may be an expression that must be evaluated prior to tag interpretation.”),
wherein the variable data corresponds to variable physical properties of the at least one virtual object[[s]] (p. 2, par. 6: “Virtual scenes are dynamically generated from virtual scene models. These virtual scene models are coded by the use of a high-level language called X-VRML. The language has higher expression power than VRML, allows customization of the virtual world generation process, enables selection of the virtual world contents, and allows access to data stored in database systems, or files. The retrieved data can influence the process of creation of the final virtual scene resulting in modification of its structure, contents, and appearance.”), and
wherein the variable physical properties of the at least one virtual object[[s]] are able to be modified (p. 5, par. 8: “The presence of geometry in the class definition influences the concept of class inheritance. Geometry can be modified by a subclass in the same way as attributes or methods in conventional object-oriented systems.”).
Walczak fails to teach wherein at least one of the variable physical properties comprises a range of values and a probability function for the range of values.
Detry teaches wherein at least one of the variable physical properties comprises a range of values and a probability function for the range of values (p. 3, par. 9: “Features correspond to hidden nodes of the network. When a model is associated to a scene (during learning or instantiation), the pose of feature i in that scene will be represented by the probability density function of a random variable Xi, effectively linking feature i to its instances.”).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to include the probability function with a range of values demonstrated by Detry in the XML-based modeling of Walczak, as both are in the field of endeavor of three-dimensional modeling. Doing so would allow for autonomous learning and probabilistic inference of the model, which would allow for more dynamic generation of virtual scenes like what Walczak seeks to enable.
Regarding claim 2, Walczak and Detry teach the method of claim 1. Walczak further teaches wherein the constant data and the variable data correspond[[s]] to a 3D scene description (p. 7, par. 6: “These parameters are passed to an X-VRML model that reads appropriate data from the database, formats them accordingly to the selected visualization method and returns them to the user in the form of a 3D virtual scene.”).
Regarding claim 3, Walczak and Detry teach the method of claim 1. Detry further teaches wherein the probability function for the at least one variable physical property is chosen from the group consisting of a Gaussian distribution (p.4, par. 13: “This kernel corresponds to a Gaussian-like distribution on SO(3).”), a Poisson distribution, a Delta function, a discrete probability distribution, a continuous probability distribution (p. 4, par. 7: “The continuous density function is accessed by assigning a kernel function to each particle, a technique generally known as kernel density estimation [36].”), or a conditional probability distribution.
Regarding claim 4, Walczak and Detry teach the method of claim 1. Walczak further teaches wherein the range of values is based on a user-defined range for the at least one variable physical property (p. 2, par. 7: “An architecture of a model-based virtual reality system is presented in Figure 1d. Every time a user wants to access a virtual world or a selected part of it, a request is sent to the server. The server reads the virtual world model, interprets it, and creates “on-the-fly” an appropriate description of the virtual scene. The user can use such a virtual scene in exactly the same way as a “standard” virtual scene stored in a file. During the process of virtual scene generation, however, multiple factors may be taken into consideration to influence its final form. These factors include selection criteria, user preferences – provided by the user or taken from a data repository, user privileges, creation method, and up-to-date data read from one or several databases.”.
Detry further teaches wherein the probability function is based on a user-defined function for the at least one variable physical property (p. 5, par. 6: “Inference is preceded by the definition of evidence for all features (hidden nodes) of the model (Sections 3 and 4.2). Once these priors have been defined, inference can be carried out with any applicable algorithm.”).
Regarding claim 13, Walczak teaches a non-transitory computer-readable medium comprising a computer program (p. 3, par. 6: “The same X-VRML program can be used to generate VRML descriptions on different levels of details or with different contents (e.g., depending on user privileges).”). Walczak and Detry further teach wherein the computer program, when executed on a processor, performs the method as claimed in claim 1 (as above).
Claim 14 is substantially similar to claim 1, and differs primarily in that it teaches an apparatus rather than a method. As such, it is rejected on a similar basis as claim 1.
Claim 15 is substantially similar to claim 2, and differs only in that it derives from claim 14 rather than claim 1. As such, it is rejected on a similar basis as claim 2.
Claim 16 is substantially similar to claim 3, and differs only in that it derives from claim 14 rather than claim 1. As such, it is rejected on a similar basis as claim 3.
Claim 17 is substantially similar to claim 4, and differs only in that it derives from claim 14 rather than claim 1. As such, it is rejected on a similar basis as claim 4.
Claim(s) 10-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Walczak (“X-VRML – XML Based Modeling of Virtual Reality”), and Detry (“A Probabilistic Framework for 3D Visual Object Representation”) as applied to claim 1 above, and further in view of Bradski (US 20190094981 A1).
Regarding claim 10, Walczak and Detry teach the method of claim 1, but fail to teach obtaining a historic sensor data log from a sensor system,
wherein the historic sensor data log comprises the physical properties, and changes of the physical properties of the at least one virtual object,
wherein the physical properties, and changes of the physical properties correspond to the at least one variable element[[(s)]] of the variable data; and
modifying the range of values and/or probability functions of the at least one variable element[[(s)]] based on historic data of the range of values and/or probability functions.
Bradski teaches obtaining a historic sensor data log from a sensor system (par. 0847: “In some embodiments, as the user utilizes the wearable device, historical data about the user is being acquired and maintained, e.g., to reflect location, activity, and copies of sensor data for that user over a period of time.”),
wherein the historic sensor data log comprises the physical properties, and changes of the physical properties of the at least one virtual object (par. 0847, as above),
wherein the physical properties, and changes of the physical properties correspond to the at least one variable element(s) of the variable data (par. 0847, as above); and
modifying the range of values and/or probability functions of the at least one variable element[[(s)]] based on historic data of the range of values and/or probability functions (par. 0939: “To estimate a pose at n, the wearable system may use historical data gathered from S-poses and O-poses (n−1, n−2, n−3, etc.). The pose at n is then used to project fiducials into the image captured at n to create an image mask from the projection. The wearable system extracts points from the masked regions and calculates the O-pose from the extracted points and mature world fiducials.”).
It would have been obvious to one familiar in the art to incorporate the 3D virtual object representation techniques of Walczak and Detry into the virtual reality system of Bradski. Both are well-known in the art, and would prove obviously beneficial to storing the massive amounts of three-dimensional environmental data necessitated by the system of Bradski.
Regarding claim 11, Walczak, Detry, and Bradski teach the method of claim 10. Detry further teaches wherein modifying the range of values and/or probability functions is based on the output of a machine learning algorithm (p. 2, par. 2: “The learning algorithm builds a hierarchy from a set of observations from a segmented object; the hierarchy is then used to recover the pose of the object in a cluttered scene.”),
wherein the machine learning algorithm is trained to identify patterns between the historic data of the range of values and/or probability functions and historic sensor data logs (p. 4, par. 1: “The structure of the hierarchy is reflected by the edge pattern of the network; each metafeature is thus linked to its child features.”).
Regarding claim 12, Van Dusen and Bradski teach the method of claim 10. Bradski further teaches modifying the range of values and/or the probability function based on a maximum a posteriori estimate of the historic sensor data log (par. 0939: “To estimate a pose at n, the wearable system may use historical data gathered from S-poses and O-poses (n−1, n−2, n−3, etc.). The pose at n is then used to project fiducials into the image captured at n to create an image mask from the projection.”).
Response to Arguments
Applicant’s arguments, see Remarks, filed 5/12/2026, with respect to the rejection(s) of claim(s) 1-4 and 13-18 under Van Dusen have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Walczak (“X-VRML – XML Based Modeling of Virtual Reality”), and further in view of Detry (“A Probabilistic Framework for 3D Visual Object Representation”).
Applicant’s arguments, see Remarks, filed 5/12/2026, with respect to the rejection(s) of claim(s) 5-9 under Bradski have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Bradski (US 20190094981 A1). The Examiner has re-mapped the Bradski citation as to better identify where Bradski discloses searching for real objects which correspond to virtual objects.
Applicant’s arguments with respect to claim(s) 10-12 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN A BARHAM whose telephone number is (571)272-4338. The examiner can normally be reached Mon-Fri, 8:30am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu, can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN ALLEN BARHAM/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights