DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Summary
This action is a responsive to the amendment filed on 7/28/2026.
Claim 13 has been canceled.
Claims 1-12, 14-22 are pending and have been examined.
Claims 1-12, 14-22 are rejected.
Response to Arguments
Rejection of Claims under 35 USC 103
Applicant’s Response:
Applicant submits that the cited references fail to teach the newly added limitation of:
“receive, from the second device, a selection including a feature from the description and content for display proximate to the feature”.
Examiner’s Response:
Applicant's arguments filed 7/28/26 have been fully considered but they are not persuasive. This limitation is taught by the combination of Petrov et al. (US 20230252736 A1), hereinafter Petrov, and Spivackv et al. (US 20190107991 A1), hereinafter Spivackv. The limitation states
receive, from the second device, a selection including a feature from the description and content for display proximate to the feature
In other words, receive a feature from a list and display content near that feature. Spivack (See ¶¶ [0365]-[0368]) teaches that a request for an object to be post at a location is received from another user. That object is then placed at the location. Petrov (See ¶¶ [0126], [0054]-[0055], [0127]) teaches creating a description of an area. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for Spivack’s location to be from Petrov’s description. Thus, the combination of Spivack and Petrov teaches the limitation.
Applicant’s Response:
Independent claim 16 is amended to recite "sending a description including text representing a subset of the information to a second device based on the information and the criteria." The Office action cites to Petrov as teaching, "sending a representing a subset of information to a second device based on the information and the criteria; and displaying content associated with the second device in a field of view of a real-world environment corresponding to the subset of sensor information." Nowhere does Petrov describe sending a description including text, however. At best, Petrov describes in paragraph [0115], "the metadata associated with the real-world object may include data that identifies a specific instance of the object to which the XR object corresponding to the electronic message should be attached such as images of the object, a 3D model of the object, feature descriptors of the object, or the like" and in paragraph [0140], "In some implementations, the metadata included within the subsequent electronic message indicates a representation or a model of the different real-world object (e.g., images of the object, a 3D model of the object, feature descriptors of the object, or the like). In some implementations, the metadata included within the subsequent electronic message indicates a location of the different real-world object." Nowhere does Petrov describe "sending a description including text representing a subset of the information to a second device based on the information and the criteria." For at least this reason, amended claim 16 is allowable over Kundu, Campbell, and Petrov.
Examiner’s Response:
Applicant's arguments filed 7/28/26 have been fully considered but they are not persuasive. The Applicant argues that metadata is not a description including text. Metadata is data about data in a structured format. Petrov et al. teaches
FIG. 4B shows an example data structure for the environment descriptor 445 in accordance with some implementations. As shown in FIG. 4B, the environment descriptor 445 may correspond to an N-tuple characterization vector or characterization tensor that includes a timestamp 461 (e.g., the most recent time the environment descriptor 445 was updated), object recognition information 462 associated with physical objects recognized within the physical environment 105 (e.g., based on a classification algorithm, computer vision (CV) techniques, or the like), instance segmentation information 464A associated with the physical environment 105, semantic segmentation information 464B such as labels for physical objects within the physical environment 105, SLAM information 466 associated with the physical environment 105 (e.g., a map, a mesh, a point cloud, or the like for the physical environment 105 as well as the current location of the electronic device 120 therewithin), and/or miscellaneous information 468. One of ordinary skill in the art will appreciate that the data structure for the environment descriptor 445 in FIG. 4B is merely an example that may include different information portions in various other implementations and be structured in myriad ways in various other implementations. ¶ [0084]
This limitation is taught by the combination of Kundu et al. (US 20230298226 A1), hereinafter Kundu, and Petrov et al. (US 20230252736 A1), hereinafter Petrov. Kundu (¶[0241]) teaches sending video to a secondary device. Petrov (See ¶¶ [0126], [0054]-[0055], [0127]) teaches creating a description of an area. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to send Petrov’s description in place of video. Thus, the combination of Kundu and Petrov teaches the limitation.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-12, 14-22 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Similarly, original claims may lack written description when the claims define the invention in functional language specifying a desired result but the specification does not sufficiently describe how the function is performed or the result is achieved. For software, this can occur when the algorithm or steps/procedure for performing the computer function are not explained at all or are not explained in sufficient detail (simply restating the function recited in the claim is not necessarily sufficient). In other words, the algorithm or steps/procedure taken to perform the function must be described with sufficient detail so that one of ordinary skill in the art would understand how the inventor intended the function to be performed. See MPEP §§ 2163.02 and 2181, subsection IV.
The level of detail required to satisfy the written description requirement varies depending on the nature and scope of the claims and on the complexity and predictability of the relevant technology. Ariad, 598 F.3d at 1351, 94 USPQ2d at 1172; Capon v. Eshhar, 418 F.3d 1349, 1357-58, 76 USPQ2d 1078, 1083-84 (Fed. Cir. 2005). Computer-implemented inventions are often disclosed and claimed in terms of their functionality. For computer-implemented inventions, the determination of the sufficiency of disclosure will require an inquiry into the sufficiency of both the disclosed hardware and the disclosed software due to the interrelationship and interdependence of computer hardware and software. The critical inquiry is whether the disclosure of the application relied upon reasonably conveys to those skilled in the art that the inventor had possession of the claimed subject matter as of the filing date. Vasudevan Software, Inc. v. MicroStrategy, Inc., 782 F.3d 671, 682. 114 USPQ2d 1349, 1356 (citing Ariad Pharm., Inc. V. Eli Lilly & Co, 598 F.3d 1336, 1351, 94 USPQ2d 1161, 1172 (Fed. Cir. 2010) in the context of determining possession of a claimed means of accessing disparate databases).
When examining computer-implemented functional claims, examiners should determine whether the specification discloses the computer and the algorithm (e.g., the necessary steps and/or flowcharts) that perform the claimed function in sufficient detail such that one of ordinary skill in the art can reasonably conclude that the inventor possessed the claimed subject matter at the time of filing. An algorithm is defined, for example, as "a finite sequence of steps for solving a logical or mathematical problem or performing a task." Microsoft Computer Dictionary (5th ed., 2002). Applicant may "express that algorithm in any understandable terms including as a mathematical formula, in prose, or as a flow chart, or in any other manner that provides sufficient structure." Finisar Corp. v. DirecTV Grp., Inc., 523 F.3d 1323, 1340, 86 USPQ2d 1609, 1623 (Fed. Cir. 2008) (internal citation omitted). It is not enough that one skilled in the art could write a program to achieve the claimed function because the specification must explain how the inventor intends to achieve the claimed function to satisfy the written description requirement. See, e.g., Vasudevan Software, Inc. v. MicroStrategy, Inc., 782 F.3d 671, 681-683, 114 USPQ2d 1349, 1356, 1357 (Fed. Cir. 2015) (reversing and remanding the district court’s grant of summary judgment of invalidity for lack of adequate written description where there were genuine issues of material fact regarding "whether the specification show[ed] possession by the inventor of how accessing disparate databases is achieved"). If the specification does not provide a disclosure of the computer and algorithm in sufficient detail to demonstrate to one of ordinary skill in the art that the inventor possessed the invention a rejection under 35 U.S.C. 112(a) or pre-AIA 35 U.S.C. 112, first paragraph, for lack of written description must be made. For more information regarding the written description requirement, see MPEP § 2162- § 2163.07(b). If the specification does not provide a disclosure of sufficient corresponding structure, materials, or acts that perform the entire claimed function of a means- (or step-) plus- function limitation in a claim under 35 U.S.C. 112(f) or the sixth paragraph of pre-AIA 35 U.S.C. 112, "the applicant has in effect failed to particularly point out and distinctly claim the invention" as required by the 35 U.S.C. 112(b) [or the second paragraph of pre-AIA 35 U.S.C. 112 ]. In re Donaldson Co., 16 F.3d 1189, 1195, 29 USPQ2d 1845, 1850 (Fed. Cir. 1994) (en banc). A rejection under 35 U.S.C. 112(b) or the second paragraph of pre-AIA 35 U.S.C. 112 must be made in addition to the written description rejection. See also MPEP § 2181, subsection II.B.2(a). MPEP 2161.01
Claims 1, 8, and 16 require a senor (camera) data being converted into text description to be sent. The specification provides no means (algorithm, flowchart, the necessary steps, etc.) that one of ordinary skill in the art can reasonably conclude that the inventor possessed the claimed subject matter at the time of filing. The only support of this limitation is found in paragraph [0062]. It only provides a desired result but the specification does not sufficiently describe how the function is performed or the result is achieved.
In other examples, however, sensor filtering module 312 may identify real-world environmental features in the sensor information and send a text-based list of real-world environmental features to the second device. ¶ [0062]
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 2 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends.
Claim 2 states “display the content in a field of view proximate to the feature”. Claim 1 already states “display the content proximate to the feature in a field of view including the real-world environment around the computing device”. Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7 are rejected under 35 U.S.C. 103 as being unpatentable over Kundu et al. (US 20230298226 A1) and further in view of Campbell et al. (US 20220255763 A1) and Petrov et al. (US 20230252736 A1) and Spivackv et al. (US 20190107991 A1).
As to claim 1, Kundu et al. teaches a computing device comprising: a processor; a memory configured with instructions to (See ¶ [0220], Teaches that Computing platform(s) 1502 may include electronic storage 1528, one or more processors 1530, and/or other components):
filter information from the sensor based on the criterion to generate a subset of the information (See ¶¶ [0092]-[0093], [0085], [0091], Teaches that detection modules 905 may be run on the segmented data to determine whether they cross a privacy score threshold. For example, detection modules 905 may use features such as where faces are looking (directly at the camera or not), media metadata such as a location which can map to the user's house, etc. In addition, the privacy score may be calculated and analyzed for each segmented element. Once the privacy score value crosses a privacy threshold, the media may be fixed. However, detection modules 905 may also check privacy settings 907 to determine whether a particular element should remain unmodified. This process may occur locally with a companion app, or remotely on a server. The user may be required to give consent for the server processing in order to protect other people's privacy. GAN models 904 may be used to generate synthetic data to replace the privacy invasive portions of the media. Thus, a synthetic face may be displayed, rather than a blur or a black box. This process may also be applied to other forms of media, such as videos, audio (e.g., voice), etc. At step 802 one or more of the systems described herein may receive media content. The systems described herein may perform step 802 in a variety of ways. In one example, the media content may be received and stored locally. A user may capture media using a computing device 901, which may correspond to a user device such as a mobile device, laptop, etc. and/or may correspond to an artificial reality device, such as augmented reality glasses, augmented reality watch, etc.).
However, it does not expressly teach the details of receive a criterion for filtering, the criterion relating to a sensor recording a real-world environment around the computing device; send a description representing the subset of the information to a second device; and receive, from the second device, a selection including a feature from the description and content for display proximate to the feature; and a display configured to display the content proximate to the feature in a field of view including the real-world environment around the computing device.
Campbell et al., from analogous art, teaches receive a criterion for filtering, the criterion relating to a sensor recording a real-world environment around the computing device (See ¶¶ [0083], [0077], [0155], Teaches that a user device 102 receives privacy settings associated with a user. The user device 102 can encrypt the new or updated privacy settings. The privacy settings can include new settings or updates to previously received settings. A user associated with a user device 102 can also provide user privacy settings, or privacy settings. The user has control over the user's own user data, or environment data, generated by their device(s). The user also has control over what user data is accessible from other sources (e.g., (e.g., third parties, other devices, the user herself, or any other data available). The data may include data (a) captured from sensors (which may be, e.g., operatively coupled to the frame 1012 or otherwise attached to the user 1004), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, or gyros;).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Campbell et al. into Kundu et al. in order to permit the user to control the protection level(s) for this information, so that the user is in control of the data collection and/or sharing of user data based on the user's personal privacy preferences (See Campbell et al. ¶ [0022]).
However, it does not expressly teach the details of send a description representing the subset of the information to a second device; and receive, from the second device, a selection including a feature from the description and content for display proximate to the feature; and a display configured to display the content proximate to the feature in a field of view including the real-world environment around the computing device.
Petrov et al., from analogous art, teaches send a description representing the subset of the information to a second device (See ¶¶ [0126], [0054]-[0055], [0127], Teaches that the method 700 includes obtaining (e.g., receiving, retrieving, or determining/generating) a physical environment descriptor associated with the current FOV of the physical environment. In some implementations, as represented by block 742, the physical environment descriptor includes at least one of object recognition information, instance segmentation information, semantic segmentation information, SLAM information, or the like associated with the current FOV of the physical environment.);
and a display configured to display the content proximate to the feature in a field of view including the real-world environment around the computing device (See ¶¶ [0128], [0054]-[0055], Teaches that the method 700 includes determining whether the current FOV of the physical environment includes the real-world object based on the physical environment descriptor. In accordance with a determination that the current FOV of the physical environment includes the real-world object, the method 700 continues to block 760. In accordance with a determination that the current FOV of the physical environment does not include the real-world object, the method 700 continues to block 730 (e.g., the computing system continues obtaining images(s) associated with the current FOV of the physical environment. As one example, with reference to FIGS. 2 and 4C, in response to determining that the electronic message is attached to or associated with the real-world object, the computing system or a component thereof (e.g., the surfacer engine 439) is configured to determine whether a current FOV of the physical environment 105 includes the real-world object.).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Petrov et al. into the combination of Kundu et al and Campbell et al. in order to obtain (e.g., receive, retrieve, or determine/generate) an environment descriptor based on the input data and update the environment descriptor over time (See Petrov et al. ¶ [0043]).
However, it does not expressly teach the details of receive, from the second device, a selection including a feature from the description and content for display proximate to the feature.
Spivack et al., from analogous art, teaches receive, from the second device, a selection including a feature from the description and content for display proximate to the feature (See ¶¶ [0365]-[0368], Teaches that a request for the object to be posted in the augmented reality environment for a given user. The object (e.g., VOB) can include a message, a chat, an email, a note, a post, a news item, or any other types of VOBs as disclosed herein. The given user is generally a recipient user which can include multiple recipient users or user groups or organizations or corporations. In general, in accordance with the request, the virtual object is adjusted to be world locked or user locked by preference of the given user. In process 1002, the object is world-locked. For example, the VOB can be fixed or locked to a geolocation which can include a place or a region. In process 1006, the object is associated with a given location in the real world environment, in world locking the object. In this manner the VOB can be affixed to or sticky at the physical location to which it is fixed. The VOB can also be fixed to multiple physical locations. For instance, the object can be locked to one or more locations where the recipient(s), friends or larger group can see when they are at those locations. For example, a virtual object that is a private note can be posted as a world-locked object—for example the VOB can be sent to a user Bob such that it appears in the doorway of their house in a locked position as a sticky note or other type of AR message to Bob.).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Spivack et al. into the combination of Kundu et al and Campbell et al. and Petrov et al. in order to facilitate interaction with a virtual billboard associated with a physical location in the real world environment (See Spivack et al. ¶ [0052]).
As to claim 2, the combination of Kundu et al and Campbell et al. and Petrov et al. and Spivack et al. teaches the device according to claim 1 above. However, it does not expressly teach the details of wherein the memory is further configured with instructions to: display the content in a field of view proximate to the feature.
Petrov et al., from analogous art, teaches wherein the memory is further configured with instructions to: display the content in a field of view proximate to the feature (See ¶¶ [0128], [0054]-[0055], Teaches that the method 700 includes determining whether the current FOV of the physical environment includes the real-world object based on the physical environment descriptor. In accordance with a determination that the current FOV of the physical environment includes the real-world object, the method 700 continues to block 760. In accordance with a determination that the current FOV of the physical environment does not include the real-world object, the method 700 continues to block 730 (e.g., the computing system continues obtaining images(s) associated with the current FOV of the physical environment. As one example, with reference to FIGS. 2 and 4C, in response to determining that the electronic message is attached to or associated with the real-world object, the computing system or a component thereof (e.g., the surfacer engine 439) is configured to determine whether a current FOV of the physical environment 105 includes the real-world object.).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Petrov et al. into the combination of Kundu et al and Campbell et al. and Petrov et al. and Spivack et al. in order to obtain (e.g., receive, retrieve, or determine/generate) an environment descriptor based on the input data and update the environment descriptor over time (See Petrov et al. ¶ [0043]).
As to claim 3, the combination of Kundu et al and Campbell et al. and Petrov et al. and Spivack et al. teaches the device according to claim 1 above. Kundu et al. further teaches wherein the sensor includes a global positioning system (See ¶ [0300], Teaches that These sensors may include cameras, IR sensors, heat sensors, motion sensors, GPS receivers, or in some cases, sensors that detect a user's eye movements),
the criterion includes an exclusion zone, and filtering the information based on the criterion further comprises filtering the information to remove data within the exclusion zone to generate the subset of the information (See ¶ [0235], Teaches that privacy settings may allow a user to specify one or more geographic locations from which objects can be accessed. Access or denial of access to the objects may depend on the geographic location of a user who is attempting to access the objects. As an example, and not by way of limitation, a user may share an object and specify that only users in the same city may access or view the object. As another example and not by way of limitation, a first user may share an object and specify that the object is visible to second users only while the first user is in a particular location. If the first user leaves the particular location, the object may no longer be visible to the second users. As another example and not by way of limitation, a first user may specify that an object is visible only to second users within a threshold distance from the first user. If the first user subsequently changes location, the original second users with access to the object may lose access, while a new group of second users may gain access as they come within the threshold distance of the first user.).
As to claim 4, the combination of Kundu et al and Campbell et al. and Petrov et al. and Spivack et al. teaches the device according to claim 1 above. Kundu et al. further teaches wherein the sensor includes a global positioning system (See ¶ [0300], Teaches that These sensors may include cameras, IR sensors, heat sensors, motion sensors, GPS receivers, or in some cases, sensors that detect a user's eye movements),
the criterion includes an inclusion zone, and filtering the information based on the criterion further comprises filtering the information to remove data outside the inclusion zone to generate the subset of information (See ¶ [0235], Teaches that privacy settings may allow a user to specify one or more geographic locations from which objects can be accessed. Access or denial of access to the objects may depend on the geographic location of a user who is attempting to access the objects. As an example, and not by way of limitation, a user may share an object and specify that only users in the same city may access or view the object. As another example and not by way of limitation, a first user may share an object and specify that the object is visible to second users only while the first user is in a particular location. If the first user leaves the particular location, the object may no longer be visible to the second users. As another example and not by way of limitation, a first user may specify that an object is visible only to second users within a threshold distance from the first user. If the first user subsequently changes location, the original second users with access to the object may lose access, while a new group of second users may gain access as they come within the threshold distance of the first user.).
As to claim 5, the combination of Kundu et al and Campbell et al. and Petrov et al. and Spivack et al. teaches the device according to claim 1 above. Kundu et al. further teaches wherein the sensor includes at least one of a lidar or a camera (See ¶¶ [0290], [0071], Teaches that augmented-reality system 1800 and/or virtual-reality system 1900 may include one or more optical sensors, such as two-dimensional (2D) or 3D cameras, structured light transmitters and detectors, time-of-flight depth sensors, single-beam or sweeping laser rangefinders, 3D LiDAR sensors, and/or any other suitable type or form of optical sensor. Image frames may be captured by the camera of a device at fixed intervals (e.g., every millisecond, every ten milliseconds, every second, etc.) and/or in response to user input (e.g., pressing and/or holding a “record” or “capture” button).),
the criteria includes at least one exclusion feature, and filtering the information based on the criteria further comprises filtering the information to remove data related to the at least one exclusion feature to generate the subset of information (See ¶¶ [0143], [0144], Teaches that the disclosed systems and methods may enable an owner or user of an AR system to share a live preview of captured data with bystanders. In some embodiments, the disclosed systems and methods may send the live preview from the AR system to a companion application running on another device (e.g., a smartphone or smartwatch) so that the bystanders can see how privacy is being protected prior to capture or recording. The disclosed systems and methods may enable bystanders to hold a visual pattern (e.g., a QR code) in front of an AR system's camera to filter recently captured photos/videos and/or to blur faces that match with the bystander or the bystanders' companions.).
As to claim 6, the combination of Kundu et al and Campbell et al. and Petrov et al. and Spivack et al. teaches the device according to claim 1 above. Kundu et al. further teaches wherein the subset of the information includes at least one of: an object, a surface, a context, a weather type, an event, a person, or a location (See ¶¶ [0087], [0145] Teaches that a privacy portion of a media content may correspond to faces, certain text (e.g., addresses, license plate numbers, etc.) recognizable objects, private locations (e.g., a person's bedroom, office, etc.) that are captured in the media content. The systems described herein may perform step 806 in a variety of ways. In one example, detecting the privacy portion may be based on a privacy score applied to each segmented element. For instance, the privacy score may be based on at least one of a face orientation, a media metadata, a location, or user settings. The user settings may include user selected persons or objects that are not considered private. The disclosed systems may blur faces in captured images or videos unless the person who is in the captured images or video has given permission to the system not to do so. For a person that wants to be captured by a specific AR system (or a category of these capturing systems), the disclosed systems may enable the person to grant permission through some kind of interface (e.g., using a face-identification system). If permissions are tied to a particular face identity (or something similar with encrypted information about the person's facial features), then the disclosed systems may guarantee that the identity of the one in the capture is the same as the one who grants permission.).
As to claim 7, the combination of Kundu et al and Campbell et al. and Petrov et al. and Spivack et al. teaches the device according to claim 1 above. Kundu et al. further teaches wherein the sensor comprises any combination of: a camera, a microphone, an inertial measurement unit, a global positioning system, or a lidar (See ¶¶ [0290], [0071], [0300], Teaches that augmented-reality system 1800 and/or virtual-reality system 1900 may include one or more optical sensors, such as two-dimensional (2D) or 3D cameras, structured light transmitters and detectors, time-of-flight depth sensors, single-beam or sweeping laser rangefinders, 3D LiDAR sensors, and/or any other suitable type or form of optical sensor. Image frames may be captured by the camera of a device at fixed intervals (e.g., every millisecond, every ten milliseconds, every second, etc.) and/or in response to user input (e.g., pressing and/or holding a “record” or “capture” button). These sensors may include cameras, IR sensors, heat sensors, motion sensors, GPS receivers, or in some cases, sensors that detect a user's eye movements).
Claims 8-9, 11, 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Kundu et al. (US 20230298226 A1) and further in view of Spivackv et al. (US 20190107991 A1) and Petrov et al. (US 20230252736 A1).
As to claim 8, Kundu et al. teaches a computing device comprising: a processor; and a memory configured with instructions to (See ¶ [0220], Teaches that Computing platform(s) 1502 may include electronic storage 1528, one or more processors 1530, and/or other components):
receive a description representing a subset of information from a first device, the subset of the information being based on a sensor recording a real-world environment proximate to a first device and criterion for filtering (See ¶¶ [0092]-[0093], [0143], [0144], Teaches that detection modules 905 may be run on the segmented data to determine whether they cross a privacy score threshold. For example, detection modules 905 may use features such as where faces are looking (directly at the camera or not), media metadata such as a location which can map to the user's house, etc. In addition, the privacy score may be calculated and analyzed for each segmented element. Once the privacy score value crosses a privacy threshold, the media may be fixed. However, detection modules 905 may also check privacy settings 907 to determine whether a particular element should remain unmodified. This process may occur locally with a companion app, or remotely on a server. The user may be required to give consent for the server processing in order to protect other people's privacy. GAN models 904 may be used to generate synthetic data to replace the privacy invasive portions of the media. Thus, a synthetic face may be displayed, rather than a blur or a black box. This process may also be applied to other forms of media, such as videos, audio (e.g., voice), etc. The disclosed systems and methods may enable an owner or user of an AR system to share a live preview of captured data with bystanders. In some embodiments, the disclosed systems and methods may send the live preview from the AR system to a companion application running on another device (e.g., a smartphone or smartwatch) so that the bystanders can see how privacy is being protected prior to capture or recording. The disclosed systems and methods may enable bystanders to hold a visual pattern (e.g., a QR code) in front of an AR system's camera to filter recently captured photos/videos and/or to blur faces that match with the bystander or the bystanders' companions.).
However, it does not expressly teach the details of receive a description representing a subset of information from a first device; the description including a feature; receive a selection including the feature and content generated by a user for display proximate to the feature; and send the content to the first device for display in proximity to the feature.
Spivack et al., from analogous art, teaches receive a selection including the feature (See ¶¶ [0313], [0307]-[0308], [0365]-[0368], Teaches that in 602 the user can select the text icon to create or design a billboard with text. In 604, the user can specify the text, and in 606 the user can apply a design or style (shape, color) to the text and the billboard. In 608, the user can select to share the virtual billboard object with friends or to save it to a current physical location. In 610, the virtual billboard can be sent or placed at the physical location. FIG. 6B graphically depicts additional example user interfaces 620 for creating, posting and/or sharing a virtual object s having multimedia content in accordance with embodiments of the present disclosure.);
receive content generated by a user for display proximate to the feature (See ¶¶ [0313], [0307]-[0308], [0365]-[0368], Teaches that in 602 the user can select the text icon to create or design a billboard with text. In 604, the user can specify the text, and in 606 the user can apply a design or style (shape, color) to the text and the billboard. In 608, the user can select to share the virtual billboard object with friends or to save it to a current physical location. In 610, the virtual billboard can be sent or placed at the physical location. FIG. 6B graphically depicts additional example user interfaces 620 for creating, posting and/or sharing a virtual object s having multimedia content in accordance with embodiments of the present disclosure.);
receive a selection including the feature and content generated by a user for display proximate to the feature (See ¶¶ [0313]-[0314], [0416], Teaches that in 602 the user can select the text icon to create or design a billboard with text. In 604, the user can specify the text, and in 606 the user can apply a design or style (shape, color) to the text and the billboard. In 608, the user can select to share the virtual billboard object with friends or to save it to a current physical location. In 610, the virtual billboard can be sent or placed at the physical location. FIG. 6B graphically depicts additional example user interfaces 620 for creating, posting and/or sharing a virtual object s having multimedia content in accordance with embodiments of the present disclosure. FIG. 6C graphically depicts additional example user interfaces for creating a virtual object, posting a virtual object and placing a virtual object at a physical location, in accordance with embodiments of the present disclosure. The VOB that is posted can include, a 3D object, pictures, videos, text and/or link and selected in 656 or 658. A VOB can be selected in 660 and placed in a physical location in 662. Similarly, in 664 a photo can be selected and placed at the physical location in 666. In one embodiment, for close proximity (e.g., co-located user), user interaction can be achieved when 1) user can see distance in km of friends next to their friends' name on the friends list. 2) users can see friends on map (this feature could be disabled in settings). If user taps on friend icon on map, and they were close enough to them, they could activate AR mode and then search for them using a GPS arrow. Embodiments of the present disclosure include social togetherness enabled by the augmented reality environment. Social togetherness can include, for example, giving each other objects, exploring each other's space in an augmented reality world, etc.);
and send the content to the first device for display in proximity to the feature (See ¶¶ [0313]-[0314], [0416], Teaches that in 602 the user can select the text icon to create or design a billboard with text. In 604, the user can specify the text, and in 606 the user can apply a design or style (shape, color) to the text and the billboard. In 608, the user can select to share the virtual billboard object with friends or to save it to a current physical location. In 610, the virtual billboard can be sent or placed at the physical location. FIG. 6B graphically depicts additional example user interfaces 620 for creating, posting and/or sharing a virtual object S having multimedia content in accordance with embodiments of the present disclosure. FIG. 6C graphically depicts additional example user interfaces for creating a virtual object, posting a virtual object and placing a virtual object at a physical location, in accordance with embodiments of the present disclosure. The VOB that is posted can include, a 3D object, pictures, videos, text and/or link and selected in 656 or 658. A VOB can be selected in 660 and placed in a physical location in 662. Similarly, in 664 a photo can be selected and placed at the physical location in 666. In one embodiment, for close proximity (e.g., co-located user), user interaction can be achieved when 1) user can see distance in km of friends next to their friends' name on the friends list. 2) users can see friends on map (this feature could be disabled in settings). If user taps on friend icon on map, and they were close enough to them, they could activate AR mode and then search for them using a GPS arrow. Embodiments of the present disclosure include social togetherness enabled by the augmented reality environment. Social togetherness can include, for example, giving each other objects, exploring each other's space in an augmented reality world, etc).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Spivack et al. into Kundu et al. in order to facilitate interaction with a virtual billboard associated with a physical location in the real world environment (See Spivack et al. ¶ [0052]).
However, it does not expressly teach the details of receive a description representing a subset of information from a first device.
Petrov et al., from analogous art, teaches receive a description representing a subset of information from a first device (See ¶ [0126], [0054]-[0055], Teaches that the method 700 includes obtaining (e.g., receiving, retrieving, or determining/generating) a physical environment descriptor associated with the current FOV of the physical environment. In some implementations, as represented by block 742, the physical environment descriptor includes at least one of object recognition information, instance segmentation information, semantic segmentation information, SLAM information, or the like associated with the current FOV of the physical environment.);
the description including a feature (See ¶¶ [0084], [0127], Teaches that As one example, with reference to FIGS. 2 and 4A, the computing system or a component thereof (e.g., the environment analyzer 416) obtains (e.g., receives, retrieves, or determines/generates) an environment descriptor 445 based on the input data (e.g., the local sensor data 403 and the remote sensor data 405) and updates the environment descriptor 445 over time. FIG. 4B illustrates the example environment descriptor 445 including the timestamp 461 (e.g., the most recent time the environment descriptor 445 was updated), the object recognition information 462 associated with physical objects recognized within the physical environment 105 (e.g., based on a classification algorithm, computer vision (CV) techniques, or the like), the instance segmentation information 464A associated with the physical environment 105, the semantic segmentation information 464B such as labels for physical objects recognized or detected within the physical environment 105, the SLAM information 466 associated with the physical environment 105 (e.g., a map, a mesh, a point cloud, or the like for the physical environment 105 as well as the current location of the electronic device 120 therewithin), and/or miscellaneous information 468.).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Petrov et al. into the combination of Kundu et al and Spivack et al. in order to obtain (e.g., receive, retrieve, or determine/generate) an environment descriptor based on the input data and update the environment descriptor over time (See Petrov et al. ¶ [0043]).
As to claim 9, the combination of Kundu et al. and Spivack et al. Petrov et al teaches the device according to claim 8 above. However, it does not expressly teach the details of wherein receiving the content further comprises: executing a content generation module with selectable settings operable to generate the content.
Spivack et al., from analogous art, teaches wherein receiving the content further comprises: executing a content generation module with selectable settings operable to generate the content (See ¶¶ [0313], [0307]-[0308], Teaches that in 602 the user can select the text icon to create or design a billboard with text. In 604, the user can specify the text, and in 606 the user can apply a design or style (shape, color) to the text and the billboard. In 608, the user can select to share the virtual billboard object with friends or to save it to a current physical location. In 610, the virtual billboard can be sent or placed at the physical location. FIG. 6B graphically depicts additional example user interfaces 620 for creating, posting and/or sharing a virtual object s having multimedia content in accordance with embodiments of the present disclosure.).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Spivack et al. into the combination of Kundu et al. and Spivack et al. and Petrov et al. in order to facilitate interaction with a virtual billboard associated with a physical location in the real world environment (See Spivack et al. ¶ [0052]).
As to claim 11, the combination of Kundu et al and Spivack et al. and Petrov et al. teaches the device according to claim 8 above. Kundu et al. further teaches wherein receiving the description representing the subset of the information from the first device further comprises: receiving a three-dimensional rendering of static components of the subset of the information (See ¶¶ [0143], [0144], [0271] Teaches that the disclosed systems and methods may enable an owner or user of an AR system to share a live preview of captured data with bystanders. In some embodiments, the disclosed systems and methods may send the live preview from the AR system to a companion application running on another device (e.g., a smartphone or smartwatch) so that the bystanders can see how privacy is being protected prior to capture or recording. The disclosed systems and methods may enable bystanders to hold a visual pattern (e.g., a QR code) in front of an AR system's camera to filter recently captured photos/videos and/or to blur faces that match with the bystander or the bystanders' companions. The artificial-reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional (3D) effect to the viewer)).
As to claim 14, the combination of Kundu et al and Spivack et al. and Petrov et al. teaches the device according to claim 8 above. Kundu et al. further teaches wherein the subset of the information includes at least one of: an object, a surface, a context, a weather type, an event, a person, or a location (See ¶¶ [0087], [0145] Teaches that a privacy portion of a media content may correspond to faces, certain text (e.g., addresses, license plate numbers, etc.) recognizable objects, private locations (e.g., a person's bedroom, office, etc.) that are captured in the media content. The systems described herein may perform step 806 in a variety of ways. In one example, detecting the privacy portion may be based on a privacy score applied to each segmented element. For instance, the privacy score may be based on at least one of a face orientation, a media metadata, a location, or user settings. The user settings may include user selected persons or objects that are not considered private. The disclosed systems may blur faces in captured images or videos unless the person who is in the captured images or video has given permission to the system not to do so. For a person that wants to be captured by a specific AR system (or a category of these capturing systems), the disclosed systems may enable the person to grant permission through some kind of interface (e.g., using a face-identification system). If permissions are tied to a particular face identity (or something similar with encrypted information about the person's facial features), then the disclosed systems may guarantee that the identity of the one in the capture is the same as the one who grants permission.).
As to claim 15, the combination of Kundu et al and Spivack et al. and Petrov et al. teaches the device according to claim 8 above. Kundu et al. further teaches wherein the sensor comprises any combination of: a camera, a microphone, an inertial measurement unit, a global positioning system, or a lidar (See ¶¶ [0290], [0071], [0300], Teaches that augmented-reality system 1800 and/or virtual-reality system 1900 may include one or more optical sensors, such as two-dimensional (2D) or 3D cameras, structured light transmitters and detectors, time-of-flight depth sensors, single-beam or sweeping laser rangefinders, 3D LiDAR sensors, and/or any other suitable type or form of optical sensor. Image frames may be captured by the camera of a device at fixed intervals (e.g., every millisecond, every ten milliseconds, every second, etc.) and/or in response to user input (e.g., pressing and/or holding a “record” or “capture” button). These sensors may include cameras, IR sensors, heat sensors, motion sensors, GPS receivers, or in some cases, sensors that detect a user's eye movements).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Kundu et al. (US 20230298226 A1) and Spivackv et al. (US 20190107991 A1) and Petrov et al. (US 20230252736 A1) and further in view of TICHENOR et al. (US 20230162453 A1).
As to claim 10, the combination of Kundu et al. and Spivack et al. and Petrov et al. teaches the device according to claim 8 above. However, it does not expressly teach the details of wherein receiving the content further comprises: executing a content association module with options operable to, upon selection, select a feature from the description.
Spivack et al., from analogous art, teaches wherein receiving the content further comprises: executing a content association module with options operable to, upon selection, select a feature from the description (See ¶ [0107], Teaches that process 800 can receive an identification of a surface. This can be an existing surface or a surface created to accommodate a new augment. In various implementations, an artificial reality system can have created one or more surfaces. In various implementations, surfaces can be (i) synthetic (generated automatically by the artificial reality system without regard to the environment, e.g., not world-locked), (ii) semantic (detected by machine learning recognizers e.g., hands, faces, table, or other particular objects, etc.), or (iii) geometric (identified geometries in the environment e.g., floors, walls, etc.) Thus, the artificial reality system can create surfaces by defining surfaces relative to the artificial reality system position, by identifying artificial reality environment surface geometries or object types specified for creating a surface, or in response to a user defining a surface (e.g., by performing an air tap, outlining a surface with a gesture, putting a hand, controller, or other peripheral device on the surface, defining an plane in the air with her hand to be used as a surface, interacting with an object type, such as handlebars on a bike, to define a surface, etc.).).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of TICHENOR et al. into the combination of Kundu et al. and Spivack et al. and Petrov et al. in order to provide an artificial reality environment with a 2D or 3D volume, in an artificial reality environment, that can include presentation data, context, and logic (See TICHENOR et al. ¶ [0026]).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Kundu et al. (US 20230298226 A1) and Spivackv et al. (US 20190107991 A1) and Petrov et al. (US 20230252736 A1) and further in view of LADAVAC et al. (US 20230394767 A1).
As to claim 12, the combination of Kundu et al and Spivack et al. and Petrov et al. teaches the device according to claim 11 above. However, it does not expressly teach the details of wherein the three-dimensional rendering of the subset of the information is a clay rendering.
LADAVAC et al., from analogous art, teaches wherein the three-dimensional rendering of the subset of the information is a clay rendering (See ¶¶ [0030], [0059], Teaches that the mesh can be defined with sufficiently reduced complexity such that it can be processed, transmitted, and/or rendered for display quickly and stored in reduced storage space compared to other representations of surfaces and objects. For example, such a mesh is commonly used in applications such as efficient rendering of terrains, medical images, or objects modeled in a clay-like manner. Common applications include efficient rendering of terrains, medical images, or objects modeled in a clay-like manner. For example, the mesh can be used in the generation of a feature surface of a feature, such as the surface of a terrain feature (landscape, mountains, etc.), organ or other body part, digital object, etc.).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of LADAVAC et al. into the combination of Kundu et al and Spivack et al. and Petrov et al. in order to reduced complexity such that it can be processed, transmitted, and/or rendered for display quickly and stored in reduced storage space compared to other representations of surfaces and objects (See LADAVAC et al. ¶ [0030]).
Claims 16-17 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Kundu et al. (US 20230298226 A1) and further in view of Campbell et al. (US 20220255763 A1) and Petrov et al. (US 20230252736 A1).
As to claim 16, Kundu et al. teaches a method, comprising: receiving information from a sensor recording a real-world environment around the first device (See ¶¶ [0085], [0091], Teaches that , at step 802 one or more of the systems described herein may receive media content. The systems described herein may perform step 802 in a variety of ways. In one example, the media content may be received and stored locally. A user may capture media using a computing device 901, which may correspond to a user device such as a mobile device, laptop, etc. and/or may correspond to an artificial reality device, such as augmented reality glasses, augmented reality watch, etc. ).
However, it does not expressly teach the details of receiving a criterion for filtering from a first device; description including text representing a subset of the information to a second device based on the information and the criterion.
Campbell et al., from analogous art, teaches receiving a criterion for filtering from a first device (See ¶¶ [0083], [0077], [0155], Teaches that a user device 102 receives privacy settings associated with a user. The user device 102 can encrypt the new or updated privacy settings. The privacy settings can include new settings or updates to previously received settings. A user associated with a user device 102 can also provide user privacy settings, or privacy settings. The user has control over the user's own user data, or environment data, generated by their device(s). The user also has control over what user data is accessible from other sources (e.g., (e.g., third parties, other devices, the user herself, or any other data available). The data may include data (a) captured from sensors (which may be, e.g., operatively coupled to the frame 1012 or otherwise attached to the user 1004), such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, or gyros;).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Campbell et al. into Kundu et al. in order to permit the user to control the protection level(s) for this information, so that the user is in control of the data collection and/or sharing of user data based on the user's personal privacy preferences (See Campbell et al. ¶ [0022]).
However, it does not expressly teach the details of sending a description including text representing a subset of the information to a second device based on the information and the criterion.
Petrov et al., from analogous art, teaches sending a description including text representing a subset of the information to a second device based on the information and the criterion (See ¶ [0126], [0054]-[0055], [0084], [0127] Teaches that the method 700 includes obtaining (e.g., receiving, retrieving, or determining/generating) a physical environment descriptor associated with the current FOV of the physical environment. In some implementations, as represented by block 742, the physical environment descriptor includes at least one of object recognition information, instance segmentation information, semantic segmentation information, SLAM information, or the like associated with the current FOV of the physical environment. The example environment descriptor 445 including the timestamp 461 (e.g., the most recent time the environment descriptor 445 was updated), the object recognition information 462 associated with physical objects recognized within the physical environment 105 (e.g., based on a classification algorithm, computer vision (CV) techniques, or the like), the instance segmentation information 464A associated with the physical environment 105, the semantic segmentation information 464B such as labels for physical objects recognized or detected within the physical environment 105, the SLAM information 466 associated with the physical environment 105 (e.g., a map, a mesh, a point cloud, or the like for the physical environment 105 as well as the current location of the electronic device 120 therewithin), and/or miscellaneous information 468.).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Petrov et al. into the combination of Kundu et al and Campbell et al. in order to obtain (e.g., receive, retrieve, or determine/generate) an environment descriptor based on the input data and update the environment descriptor over time (See Petrov et al. ¶ [0043]).
As to claim 17, the combination of Kundu et al and Campbell et al. and Petrov et al. teaches the device according to claim 16 above. However, it does not expressly teach the details of wherein sending the description representing the subset of the information to the second device further comprises: generating a rendering of static components of the subset of the information.
Petrov et al., from analogous art, teaches of wherein sending the description representing the subset of the information to the second device further comprises: generating a rendering of static components of the subset of the information (See ¶¶ [0128], [0054]-[0055], Teaches that the method 700 includes determining whether the current FOV of the physical environment includes the real-world object based on the physical environment descriptor. In accordance with a determination that the current FOV of the physical environment includes the real-world object, the method 700 continues to block 760. In accordance with a determination that the current FOV of the physical environment does not include the real-world object, the method 700 continues to block 730 (e.g., the computing system continues obtaining images(s) associated with the current FOV of the physical environment. As one example, with reference to FIGS. 2 and 4C, in response to determining that the electronic message is attached to or associated with the real-world object, the computing system or a component thereof (e.g., the surfacer engine 439) is configured to determine whether a current FOV of the physical environment 105 includes the real-world object.).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Petrov et al. into the combination of Kundu et al and Campbell et al. and Petrov et al. in order to obtain (e.g., receive, retrieve, or determine/generate) an environment descriptor based on the input data and update the environment descriptor over time (See Petrov et al. ¶ [0043]).
As to claim 21, the combination of Kundu et al and Campbell et al. and Petrov et al. teaches the method according to claim 16 above. Kundu et al. further teaches wherein the subset of the information includes at least one of: an object, a surface, a context, a weather type, an event, a person, or a location (See ¶¶ [0087], [0145] Teaches that a privacy portion of a media content may correspond to faces, certain text (e.g., addresses, license plate numbers, etc.) recognizable objects, private locations (e.g., a person's bedroom, office, etc.) that are captured in the media content. The systems described herein may perform step 806 in a variety of ways. In one example, detecting the privacy portion may be based on a privacy score applied to each segmented element. For instance, the privacy score may be based on at least one of a face orientation, a media metadata, a location, or user settings. The user settings may include user selected persons or objects that are not considered private. The disclosed systems may blur faces in captured images or videos unless the person who is in the captured images or video has given permission to the system not to do so. For a person that wants to be captured by a specific AR system (or a category of these capturing systems), the disclosed systems may enable the person to grant permission through some kind of interface (e.g., using a face-identification system). If permissions are tied to a particular face identity (or something similar with encrypted information about the person's facial features), then the disclosed systems may guarantee that the identity of the one in the capture is the same as the one who grants permission.).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Kundu et al. (US 20230298226 A1) and Campbell et al. (US 20220255763 A1) and Petrov et al. (US 20230252736 A1) and further in view of Dougherty et al. (US 20100257252 A1).
As to claim 18, the combination of Kundu et al and Campbell et al. and Petrov et al. teaches the method according to claim 17 above. However, it does not expressly teach the details of wherein the subset of the information includes a location and the rendering is generated based on a database of geospatial data for the location.
Dougherty et al., from analogous art, teaches wherein the subset of the information includes a location and the rendering is generated based on a database of geospatial data for the location (See ¶¶ [0054]-[005], Teaches that in some embodiments, the database may comprise predetermined data such as feature descriptors and metadata associated with one or more landmarks. The predetermined data may be provided by the service provider. Additionally and optionally, the data may be user defined and transmitted by users. For example, landmarks that are not represented by pre-populated feature descriptors in the database may be represented by images provided by users. The term landmark may comprise any recognizable feature in an image, such as a textured portion of any object. For example, the blade of a windmill and the letter ‘G’ of an artist's signature in a wall painting might be two of the detected landmarks in the captured image of a room scene. When a pattern fails to be recognized by the image recognition engines, it may be determined that the pattern represents a new landmark and the user transmitted image may be used to represent the new landmark. In an embodiment, a user may decide that they desire to augment some space with content of their own choosing. For example, a user may enter an unknown area, collect information about the area such as feature descriptors, map data, and the like, and register the information in a database such that other users entering the area may then recognize the area and their place within the area. Additionally and optionally, the user or an application may choose to associate their own augmentation metadata with the area (e.g., placing virtual graffiti in the space) and make such data available to other users who may observe the area at the same or a different time. Multiple users may associate different metadata with a single area and allow the data to be accessible to different subsets of users. For example, a user may anchor some specific virtual content representing a small statue in a tavern, which may then be made visible to the user's on-line video game group when they enter the tavern while the virtual content may not be seen by any other mobile users in other video game groups. In another example, another user may have augmented the tavern with animated dancing animals. By enabling such augmentation and data sharing, the members of any type of gaming, social, or other type of group may share in the same set of common information about the tavern, its landmark descriptors, and their locations. At the same time, all users may not necessarily share in the same metadata associated with the venue.).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Dougherty et al. into the combination of Kundu et al and Campbell et al. and Petrov et al. in order to create a merged user environment (See Dougherty et al. ¶ [0003]).
Claims 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kundu et al. (US 20230298226 A1) and Campbell et al. (US 20220255763 A1 and Petrov et al. (US 20230252736 A1) and further in view of TICHENOR et al. (US 20230162453 A1).
As to claim 19, the combination of Kundu et al. and Campbell et al. and Petrov et al. teaches the method according to claim 17 above. However, it does not expressly teach the details of wherein the subset of the information is a list of environmental features.
TICHENOR et al., from analogous art, teaches wherein the subset of the information is a list of environmental features (See ¶¶ [0107], [0119], [0088] Teaches that process 800 can receive an identification of a surface. This can be an existing surface or a surface created to accommodate a new augment. In various implementations, an artificial reality system can have created one or more surfaces. In various implementations, surfaces can be (i) synthetic (generated automatically by the artificial reality system without regard to the environment, e.g., not world-locked), (ii) semantic (detected by machine learning recognizers e.g., hands, faces, table, or other particular objects, etc.), or (iii) geometric (identified geometries in the environment e.g., floors, walls, etc.) Thus, the artificial reality system can create surfaces by defining surfaces relative to the artificial reality system position, by identifying artificial reality environment surface geometries or object types specified for creating a surface, or in response to a user defining a surface (e.g., by performing an air tap, outlining a surface with a gesture, putting a hand, controller, or other peripheral device on the surface, defining an plane in the air with her hand to be used as a surface, interacting with an object type, such as handlebars on a bike, to define a surface, etc.). FIG. 11 is a conceptual diagram continuing illustrating the example 1000 of augments in the artificial reality space where the artificial reality system identifies virtual surfaces. In example 1000, the artificial reality system identifies geometric surfaces by automatically locating flat surfaces of at least a certain size. In this manner, the artificial reality system has automatically identified surfaces 1102 and 1104. The artificial reality system also identified a surface on the floor but the user instructed (not shown) the system to divide that surface into surfaces 1110 and 1112. The user also caused (not shown) the artificial reality system to create surface 1108 by placing her hand on the surface. Existing real and virtual objects located on these surfaces are automatically added to them. ).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of TICHENOR et al. into the combination of Kundu et al and Campbell et al. and Petrov et al. in order to provide an artificial reality environment with a 2D or 3D volume, in an artificial reality environment, that can include presentation data, context, and logic (See TICHENOR et al. ¶ [0026]).
As to claim 20, the combination of Kundu et al and Campbell et al. and Petrov et al. teaches the method according to claim 16 above. However, it does not expressly teach the details of further comprising: receiving content from the second device; receiving feature from the second device selected from the description; and sending the content and the feature to the first device.
TICHENOR et al., from analogous art, teaches further comprising: receiving content from the second device (See ¶ [0088], Teaches that can receive a request, associated with a manifest, for a new augment. As discussed above in relation to FIG. 5 , the manifest can specify properties and/or logic for the new augment, such as display modes (e.g., one or more context factor values that must occur for the display mode to be enabled, types of surfaces on which the augment can be placed in that display mode, a location, shape, or orientation, whether the augment is moveable, a volume for audio presentation data, etc.), a title, an ID, and augment type, an owner of the augment, an initial location for the augment, etc. In some implementations, default or inherited properties and/or logic can also be specified for the augment, such as having all augments include certain display modes that correspond to artificial reality system modes (e.g., audio only, minimized, interactive, etc.).);
receiving feature from the second device selected from the description (See ¶ [0107], Teaches that process 800 can receive an identification of a surface. This can be an existing surface or a surface created to accommodate a new augment. In various implementations, an artificial reality system can have created one or more surfaces. In various implementations, surfaces can be (i) synthetic (generated automatically by the artificial reality system without regard to the environment, e.g., not world-locked), (ii) semantic (detected by machine learning recognizers e.g., hands, faces, table, or other particular objects, etc.), or (iii) geometric (identified geometries in the environment e.g., floors, walls, etc.) Thus, the artificial reality system can create surfaces by defining surfaces relative to the artificial reality system position, by identifying artificial reality environment surface geometries or object types specified for creating a surface, or in response to a user defining a surface (e.g., by performing an air tap, outlining a surface with a gesture, putting a hand, controller, or other peripheral device on the surface, defining an plane in the air with her hand to be used as a surface, interacting with an object type, such as handlebars on a bike, to define a surface, etc.));
and sending the content and the feature to the first device (See ¶ [0092], Teaches that process 600 can perform an augment placement procedure. In some implementations, block 610 can be performed prior to block 606, e.g., setting an initial augment placement before providing the handle to the requestor, for the requestor to update the placement. The placement procedure can include setting a default location or a location specified in the request received at block 602 (e.g., a location based on the user's focus area, a location relative to the requesting entity such as the same surface a requesting augment is attached to or surface defined for the hands or face of a user associated with the request). In some implementations, the placement procedure can include making the augment invisible until the requestor selects a location.).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of TICHENOR et al. into the combination of Kundu et al and Campbell et al. and Petrov et al. in order to provide an artificial reality environment with a 2D or 3D volume, in an artificial reality environment, that can include presentation data, context, and logic (See TICHENOR et al. ¶ [0026]).
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Kundu et al. (US 20230298226 A1) and Campbell et al. (US 20220255763 A1) and Petrov et al. (US 20230252736 A1) and further in view of LADAVAC et al. (US 20230394767 A1).
As to claim 22, the combination of Kundu et al and Campbell et al. and Petrov et al. teaches the device according to claim 17 above. However, it does not expressly teach the details of wherein the rendering is a clay rendering.
LADAVAC et al., from analogous art, teaches wherein the rendering is a clay rendering (See ¶¶ [0030], [0059], Teaches that the mesh can be defined with sufficiently reduced complexity such that it can be processed, transmitted, and/or rendered for display quickly and stored in reduced storage space compared to other representations of surfaces and objects. For example, such a mesh is commonly used in applications such as efficient rendering of terrains, medical images, or objects modeled in a clay-like manner. Common applications include efficient rendering of terrains, medical images, or objects modeled in a clay-like manner. For example, the mesh can be used in the generation of a feature surface of a feature, such as the surface of a terrain feature (landscape, mountains, etc.), organ or other body part, digital object, etc.).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of LADAVAC et al. into the combination of Kundu et al and Campbell et al. and Petrov et al. in order to reduced complexity such that it can be processed, transmitted, and/or rendered for display quickly and stored in reduced storage space compared to other representations of surfaces and objects (See LADAVAC et al. ¶ [0030]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to James R Hollister whose telephone number is (571)270-3152. The examiner can normally be reached Mon - Fri 7:30 am - 4:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Philip Chea can be reached at (571) 272-3951. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
James Hollister
/J.R.H./Examiner, Art Unit 2499 10/9/26
/PHILIP J CHEA/Supervisory Patent Examiner, Art Unit 2499
