Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Claims 6-9 are withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being nonelected. During a telephone conversation with Hean Koo on 6/16/26 a provisional election was made with traverse to prosecute the invention of I, claims 1-5 and 10-15. Affirmation of this election must be made by applicant in replying to this Office action. Claims 6-9 withdrawn from further consideration by the examiner, 37 CFR 1.142(b), as being drawn to a non-elected invention.
Restriction to one of the following inventions is required under 35 U.S.C. 121:
Claims 1-5 and 10-15, drawn to a process to generate augmented reality based on scene understanding, classified in G06V 20/20.
Claims 6-9, drawn to event driven continuous machine learning, classified in G06N 20/00.
The inventions are distinct, each from the other because of the following reasons:
Inventions I and II are related as combination and subcombination. Inventions in this relationship are distinct if it can be shown that (1) the combination as claimed does not require the particulars of the subcombination as claimed for patentability, and (2) that the subcombination has utility by itself or in other combinations (MPEP § 806.05(c)). In the instant case, the combination as claimed does not require the particulars of the subcombination as claimed because event driven continuous machine learning is not a requirement of the claimed generation of augmented reality based on scene understanding. The subcombination has separate utility such as event driven continuous machine learning.
The examiner has required restriction between combination and subcombination inventions. Where applicant elects a subcombination, and claims thereto are subsequently found allowable, any claim(s) depending from or otherwise requiring all the limitations of the allowable subcombination will be examined for patentability in accordance with 37 CFR 1.104. See MPEP § 821.04(a). Applicant is advised that if any claim presented in a continuation or divisional application is anticipated by, or includes all the limitations of, a claim that is allowable in the present application, such claim may be subject to provisional statutory and/or nonstatutory double patenting rejections over the claims of the instant application.
Restriction for examination purposes as indicated is proper because both these inventions listed in this action are independent or distinct for the reasons given above and there would be a serious search and/or examination burden if restriction were not required because at least the following reason(s) apply:
the inventions have acquired a separate status in the art in view of their different classification;
the inventions have acquired a separate status in the art due to their recognized divergent subject matter; and
the inventions require a different field of search (for example, searching different classes/subclasses or electronic resources, or employing different search queries),
Applicant is reminded that upon the cancellation of claims to a non-elected invention, the inventorship must be amended in compliance with 37 CFR 1.48(b) if one or more of the currently named inventors is no longer an inventor of at least one claim remaining in the application. Any amendment of inventorship must be accompanied by a request under 37 CFR 1.48(b) and by the fee required under 37 CFR 1.17(i).
Claim Objections
Claim 1 is objected to because of the following informalities: Claim 1 recites “the one or more context-driven data elements,” and its antecedent basis is unclear. For the purposes of art rejection, the Examiner is reading the limitation as “one or more context-driven data elements.” Appropriate correction is required.
Claim 2 is objected to because of the following informalities: Claim 2 does not end with a period. Appropriate correction is required.
Claim 3 is objected to because of the following informalities: Claim 3 ends with a comma. Appropriate correction is required.
Claim 4 is objected to because of the following informalities: Claim 4 does not end with a period. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 10-15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception without significantly more.
Claim 10
Step 1: Claim 10 is directed to “a system, comprising: a processor programed to: . . .,” which is a machine, thereby meeting step 1.
Step 2A, Prong One: Claim 10 recites a “mental process” abstract idea that can be performed in the human mind or by using a pen and paper:
identify at least two data objects in the content, each data object representing a virtual object or a real world (RW) object;
the content could be a photograph or a textual description; a person could identify objects depicted in the photograph or textual description.
learn contextual data between the two data objects based at least on the content from which the two data objects were identified, the contextual data defining a context in which the two data objects appeared together in the content;
the person could interpret the context based on the photograph or textual description.
generate a linked data record comprising an identification for each of the two data objects and the learned contextual data so that identification of at least one of the data objects is sufficient to identify the linked data record; and
the person could mentally establish logical linkage between objects identified.
Step 2A, Prong Two: The following additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
(a) a system, comprising: a processor programed to
(b) access content comprising text and/or an image in a structured or unstructured format;
(c) store the linked data record in a database to be later retrieved to provide context for one or more of the two data objects, the database comprising other linked data records of other data objects, wherein the stored linked data record represents contextual data learned about the two data objects and wherein the linked data record together with the other linked data records represent contextual information of data objects learned from content.
Regarding (a), the processor, under BRI, could be interpreted as a computer or computer components. The computer is recited at a high level of generality, performing a generic computer function. This generic processor limitation is no more than mere instructions to apply the exception using a generic computer component. MPEP 2106.05(f).
Regarding (b), the additional element is insignificant extra-solution activity, specifically, mere data gathering. MPEP 2106.05(g)(3), “Mere Data Gathering” examples iii, iv and MPEP 2106.05(g)(3), “Selecting a particular data source or type of data to be manipulated:” examples ii, iii.
Regarding (c), the additional element is insignificant extra-solution activity, specifically, mere data gathering/storing. MPEP 2106.05(g)(3), “Mere Data Gathering” examples iv, v.
Step 2B: Additional elements are determined not to amount to an inventive concept after having considered them both individually and in combination; and the additional elements do not amount to significantly more than the judicial exception itself.
(a) a system, comprising: a processor programed to
(b) access content comprising text and/or an image in a structured or unstructured format;
(c) store the linked data record in a database to be later retrieved to provide context for one or more of the two data objects, the database comprising other linked data records of other data objects, wherein the stored linked data record represents contextual data learned about the two data objects and wherein the linked data record together with the other linked data records represent contextual information of data objects learned from content.
Regarding (a), the device and units, under BRI, could be interpreted as a computer or computer components. The computer is recited at a high level of generality, performing a generic computer function. This generic processor limitation is no more than mere instructions to apply the exception using a generic computer component. MPEP 2106.05(f).
Regarding (b), the additional element is insignificant extra-solution activity, specifically, mere data gathering. MPEP 2106.05(g)(3), “Mere Data Gathering” examples iii, iv and MPEP 2106.05(g)(3), “Selecting a particular data source or type of data to be manipulated:” examples ii, iii. The court has recognized similar activity, receiving/access data, as well-understood, routine, conventional. MPEP 2106.05(d).II.i.
Regarding (c), the additional element is insignificant extra-solution activity, specifically, mere data gathering/storing. MPEP 2106.05(g)(3), “Mere Data Gathering” examples iv, v and MPEP 2106.05(g)(3), “Selecting a particular data source or type of data to be manipulated:” examples ii, iii. The court has recognized similar activity, storing data, as well-understood, routine, conventional. MPEP 2106.05(d).II.iv.
Therefore, Claim 10 is rejected under 35 U.S.C. 101 for being directed to an abstract idea without significantly more.
Claim 11
Step 1: Claim 11 depends on Claim 10 and is directed to a system/machine, thereby meeting step 1.
Step 2A, Prong One: Claim 11 recites the following limitations that can be practically performed in the mind or with the aid of pen and paper:
wherein the two data objects each comprise text that represents a respective virtual object or RW object, and wherein to learn contextual data,
Step 2A, Prong Two; Step 2B:
Regarding “the processor is programmed to,” the processor, under BRI, could be interpreted as a computer or computer components. The computer is recited at a high level of generality, performing a generic computer function. This generic processor limitation is no more than mere instructions to apply the exception using a generic computer component. MPEP 2106.05(f).
Claim 12
Step 1: Claim 12 depends on Claim 10 and is directed to a system/machine, thereby meeting step 1.
Step 2A, Prong One: Claim 12 recites the following limitations that can be practically performed in the mind or with the aid of pen and paper:
wherein the two data objects each comprise images that represents a respective virtual object or RW object, and wherein to learn contextual data,
Step 2A, Prong Two; Step 2B:
Regarding “the processor is programmed to,” the processor, under BRI, could be interpreted as a computer or computer components. The computer is recited at a high level of generality, performing a generic computer function. This generic processor limitation is no more than mere instructions to apply the exception using a generic computer component. MPEP 2106.05(f).
Claim 13
Step 1: Claim 13 depends on Claim 10 and is directed to a system/machine, thereby meeting step 1.
Step 2A, Prong One: Claim 13 recites the following limitations that can be practically performed in the mind or with the aid of pen and paper:
wherein to learn contextual data,
Step 2A, Prong Two; Step 2B:
Regarding “the processor is programmed to,” the processor, under BRI, could be interpreted as a computer or computer components. The computer is recited at a high level of generality, performing a generic computer function. This generic processor limitation is no more than mere instructions to apply the exception using a generic computer component. MPEP 2106.05(f).
Claim 14
Step 1: Claim 14 depends on Claim 10 and is directed to a system/machine, thereby meeting step 1.
Step 2A, Prong One: Claim 14 recites the following limitations that can be practically performed in the mind or with the aid of pen and paper:
wherein to learn contextual data,
Step 2A, Prong Two; Step 2B:
Regarding “the processor is programmed to,” the processor, under BRI, could be interpreted as a computer or computer components. The computer is recited at a high level of generality, performing a generic computer function. This generic processor limitation is no more than mere instructions to apply the exception using a generic computer component. MPEP 2106.05(f).
Claim 15
Step 1: Claim 15 depends on Claim 10 and is directed to a system/machine, thereby meeting step 1.
Step 2A, Prong One: Claim 15 recites the following limitations that can be practically performed in the mind or with the aid of pen and paper:
trigger a learning process to learn the contextual data.
a person decides to learn.
Step 2A, Prong Two; Step 2B:
(a) wherein the processor is programmed to:
(b) receive an event-driven indication that new content comprising text and/or an image has been ingested to a data lake that stores text and/or images in a structured or unstructured format; and
Regarding (a), the processor, under BRI, could be interpreted as a computer or computer components. The computer is recited at a high level of generality, performing a generic computer function. This generic processor limitation is no more than mere instructions to apply the exception using a generic computer component. MPEP 2106.05(f).
Regarding (b), the additional element is insignificant extra-solution activity, specifically, mere data gathering. MPEP 2106.05(g)(3), “Mere Data Gathering” examples iii, iv and MPEP 2106.05(g)(3), “Selecting a particular data source or type of data to be manipulated:” examples ii, iii. The court has recognized similar activity, receiving data, as well-understood, routine, conventional. MPEP 2106.05(d).II.i.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5 are rejected under 35 U.S.C. 103 as being unpatentable over Smith et al. (US20230123933A1) in view of SHAPIRA (US 20200334463 A1).
Regarding Claim 1, Smith teaches A system, comprising: a memory (Smith ¶¶ 36, 38; Smith Fig. 2) to store an AR application (Smith ¶ 37); a processor programmed to execute the AR application (Smith ¶ 77-78) and to: generate an AR display in which one or more virtual objects (Smith Fig. 3 320) are to be overlaid onto a real world (RW) environment (Environment that includes Smith Fig. 3 310) (
Smith ¶ 37;
PNG
media_image1.png
530
522
media_image1.png
Greyscale
);
access an Augmented Unification (AU) object comprising one or more properties that define a context of the RW environment based on image recognition performed on the RW environment (
Smith teaches “real context description” and associated logical rules, mapped to Augmented Unification (AU) object, stating “The real context description may be placed in a memory location having an address or pointer that is possessed by other modules of the mixed reality process, especially a content selection module. As the context changes through time, the context determination module 526 updates the current context description. A context description may include, for example, a volumetric solidity map or model of the physical environment; environmental condition parameters; scene and object types; and scene and object identities. The priority of the description may be in the order listed, with the volumetric solidity map having the highest priority. Condition information is also high priority; condition and solidity are both needed to implement insertion of mixed reality objects in a real environment according to a consistent grammar. Type and identity information is useful for providing a more varied and enriching experience, but not as fundamental to providing a basic user experience.” Smith ¶ 52.
The context description and its associated rules unite the real environment and virtual objects.
Smith provides an example of such unification, stating “A real context description may be blended with a mixed reality context description and optionally with heuristic feedback from a particular combination of user and mixed reality session to perform a virtual content selection process. By way of example, FIG. 6 shows elements of one such selection process 600 based on the real context description 528 and other inputs. A content selection 603 may include different internal modules or components. For example, a first filter process 610 parses the real context description 528 and uses a set of logical rules (which may include fuzzy logic) to eliminate library content from consideration that is incompatible with the physical parameters of the scene. This may be conceptually understood as a first filter pass of library content.” Smith ¶ 53
Smith further discloses that the RW environment is based on image recognition, stating “Recognition 522 of particular identities of scenes and objects within them can be based . . . upon object recognition and/or image analysis software.” Smith ¶ 51. “Input from type recognition 524, condition recognition 520 and identity recognition 522 may be integrated and processed by a final real context determination module 526 that produces as output computer-readable data describing a real current context, sometime referred to herein as a context description, or a real context description.” Smith ¶ 52.),
the one or more properties comprising an object relationship between data objects, visual appearance, and/or behavior of an object relationship are based on one or more physical objects recognized from the physical environment, the context defined in the AU object having been learned from text and/or images (
“A real context description may be blended with a mixed reality context description and optionally with heuristic feedback from a particular combination of user and mixed reality session to perform a virtual content selection process. By way of example, FIG. 6 shows elements of one such selection process 600 based on the real context description 528 and other inputs. A content selection 603 may include different internal modules or components. For example, a first filter process 610 parses the real context description 528 and uses a set of logical rules (which may include fuzzy logic) to eliminate library content from consideration that is incompatible with the physical parameters of the scene. This may be conceptually understood as a first filter pass of library content. ‘Incompatible’ may mean that a set of content cannot possibly comply with a specified grammar for the mixed reality session, or may have any other desired meaning so long as it results in a filtering of available content sets. It should be appreciated that ranking of content sets is equivalent to filtering.” Smith ¶ 53.
The disclosed compatibility is an example of object relationship between data objects, between data objects associated with virtual objects and physical objects, which is also an example of behavior/existence of virtual object relationship based on physical objects.
The context has been learned from images through object recognition. Smith ¶ 52.);
identify a virtual object and one or more characteristics of the virtual object based on the AU object (
Smith teaches identifying/filtering virtual objects to display in a mixed reality, stating “A real context description may be blended with a mixed reality context description and optionally with heuristic feedback from a particular combination of user and mixed reality session to perform a virtual content selection process. By way of example, FIG. 6 shows elements of one such selection process 600 based on the real context description 528 and other inputs. A content selection 603 may include different internal modules or components. For example, a first filter process 610 parses the real context description 528 and uses a set of logical rules (which may include fuzzy logic) to eliminate library content from consideration that is incompatible with the physical parameters of the scene. This may be conceptually understood as a first filter pass of library content. ‘Incompatible’ may mean that a set of content cannot possibly comply with a specified grammar for the mixed reality session, or may have any other desired meaning so long as it results in a filtering of available content sets. It should be appreciated that ranking of content sets is equivalent to filtering.” Smith ¶ 53.
Smith teaches identifying/determined appearance or behavioral characteristics, e.g., celebratory v. adversarial, stating “A third filter process 614 may re-rank or further filter the output of the second process 612, based on a current session context. For example, if the user has just finished dispatching a difficult computer-generated adversary, the process 614 may prioritize celebratory characters or objects over adversarial ones. The third process 614 selects an object set based on a final ranking, which is inserted into the mixed reality process. The behavior of the inserted object and its eventual exit from the mixed reality process may be predetermined, or may be managed in a context-responsive manner by a “life” module 616, using session context and any changes in the real context description 528 to determine behavior and other temporal changes in object characteristics, including when and how the mixed reality object is to be removed from the current session.” Smith ¶ 55.);
define a behavior of the virtual object with respect to the physical environment based on the one or more context-driven data elements (
Smith teaches context-driven data elements, mapped to changes in real context description, stating “The behavior of the inserted object and its eventual exit from the mixed reality process may be predetermined, or may be managed in a context-responsive manner by a ‘life’ module 616, using session context and any changes in the real context description 528 to determine behavior and other temporal changes in object characteristics, including when and how the mixed reality object is to be removed from the current session.” Smith ¶ 55. “For example, the object’s appearance and skin may be generated or altered based on an algorithm that takes as inputs context information and character components, and outputs a character or other object that is customized for the current context in both appearance and behavior.” Smith ¶ 40.);
receive a virtual object to augment the electronic display and one or more permissible actions that can be used based on contextual data (
[BRI on the Record]
With respect to claimed “permissible actions,” the Examiner is reading the limitation to mean actions, e.g., by a virtual object, a user, or anyone or anything, that is permitted by anyone or anything. The claim and the specification does not provide clear guidance. The Examiner suggests that the limitation could be further clarified by specifying actions by whom/what and permission by whom/what.
[Mapping Analysis]
“For example, the object’s appearance and skin may be generated or altered based on an algorithm that takes as inputs context information and character components, and outputs a character or other object that is customized for the current context in both appearance and behavior. . . .Inserting a mixed-reality animation of an un-identifiable baseball player when the sensors indicate that the user is located in or near a baseball diamond is an example of generic relevance. Inserting an animated historical or public figure when the user 401 is located in a place of significance to the figure is an example of specific relevance.” Smith ¶ 40. “The behavior of the inserted object and its eventual exit from the mixed reality process may be predetermined, or may be managed in a context-responsive manner by a “life” module 616, using session context and any changes in the real context description 528 to determine behavior and other temporal changes in object characteristics, including when and how the mixed reality object is to be removed from the current session.” Smith ¶ 40. “As the real environment and session context evolve, the virtual object’s parameters may evolve accordingly until a behavior parameter removes the virtual object from the session entirely. Until then, the virtual object may move about and react to environmental factors and user input in various ways.” Smith ¶ 94.
Smith clearly discloses the behaviors of the virtual object. Further, even the displaying of the “appearance” of a virtual object is a permitted action by the computing system.),
the virtual object and the one or more permissible actions being retrieved based on Smith ¶¶ 40, 94. As explained for the earlier limitation, permitted actions of all kinds and virtual objects are obtained based on the context of physical objects, relations among physical objects and/or physical and virtual object combination.);
update the electronic display to include the virtual object (
Smith teaches updating the displaying of virtual objects, stating “The behavior of the inserted object and its eventual exit from the mixed reality process may be predetermined, or may be managed in a context-responsive manner by a ‘life’ module 616, using session context and any changes in the real context description 528 to determine behavior and other temporal changes in object characteristics, including when and how the mixed reality object is to be removed from the current session.” Smith ¶ 55. “For example, the object’s appearance and skin may be generated or altered based on an algorithm that takes as inputs context information and character components, and outputs a character or other object that is customized for the current context in both appearance and behavior.” Smith ¶ 40.
“Herein, an ‘augmented reality device’ may be a device capable of representing augmented reality and may include, for example, not only ‘augmented reality glasses in the shape of glasses worn by a user on his/her face, but also a head-mounted display (HMD) apparatus worn by a user on his/her head, or an augmented reality helmet.” Smith ¶ 31. ); and
cause an interaction between the physical object and the virtual object based on the one or more permissible actions to be displayed in the electronic display (“These examples illustrate that one of the functions of mixed media grammar can be to maintain a defined geometric relationship between a reference object in the real scene and related objects experienced only in VR or AR.” Smith ¶ 68.
Smith provides more examples, stating “For example, a mixed reality AR application may detect that the user is viewing a video billboard or other image display appearing in the user’s real environment, and configure the mixed reality content based on the content appearing on the real display in some way. For example, the mixed reality application may be configured to replace the content appearing on a real video display or static image display with a substitute video or static image in the mixed reality session. In an alternative, or in addition, the mixed reality application may be configured to cause a virtual object or character that relates to content appearing on the real display to appear in the mixed reality session. For example, if the real content includes advertising for a particular product, the application may insert an animated mascot character for the product (or for a competing product) in the mixed reality session. For further example, if the real content depicts a particular scene, object, theme or character, the application may insert virtual content that enhances, parodies, or contradicts the real content.” Smith ¶ 30.).
Smith does not explicitly disclose the virtual object and the one or more permissible actions being retrieved from an application-specific database that stores unification data relating to a plurality of physical objects for which context, relationships between objects, and permitted actions have been learned from training data comprising images and/or text.
SHAPIRA teaches the virtual object and the one or more permissible actions being retrieved from an application-specific database that stores unification data relating to a plurality of physical objects (Shapira Fig. 1 (1) “Recognize Plurality of Objects”) for which context (Shapira Fig. 1 (2) “Spatial Context”), relationships between objects (Shapira Fig. 1 (2) “Position Relationship Between Plurality of Recognized Objects”), and permitted actions (permitted displaying at “preset display position” Shapira ¶ 51; Smith ¶¶ 30, 68.) have been learned from training data comprising images and/or text (
Shapira teaches training data including images and text, stating “The image classifier 138A may be trained to perform image recognition on a training images corpus 156, which may include labeled images. Each labeled image in the training images corpus 156 may indicate an item (such as a person, animal, thing, etc.) that is in the image. In some examples, training images corpus 156 may include additional features for training such as object segmentation (to be able to distinguish multiple objects in an image), recognition in context to support contextual image recognition, and/or other features. Examples of the training images corpus 156 may include the MNIST dataset, MS-COCO dataset, ImageNet dataset, Open Images Dataset, CIFAR-10 dataset, CIFAR-100 dataset, and/or other image training datasets.” Shapira ¶ 32.
“Alternatively, or additionally, such location identification may be driven by an image classifier trained to recognize certain locations (such as rooms in a home, retail locations such as a grocery or electronics store, and/or other types of locations). In this case, the image classifier may be trained based on a training images corpus that includes images that are labeled according to types of locations to be recognized (such as labeled images of kitchens).” Shapira ¶ 15.
Shapira teaches application-specific database, mapped to “repository,” stating “The system repository 150 may store various data used by the system 100. For example, the system repository 150 may store the narratives 152, the decision rules 154, the training images corpus 156, and/or other data. Each of data stored in the system repository 150 may be in individual storage repositories or combined together. For example, the system repository may include a repository for the narratives 152, a repository for the decision rules 154, and a repository for the training images corpus 156.” Shapira ¶ 43; also see ¶ 65.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Shapira’s database and AI model with Smith. One of ordinary skill in the art would be motivated to quickly and easily access data and to make the system easier/faster/cheaper to develop based on AI.
Regarding Claim 2, Smith in view of Shapira teaches The system of claim 1, wherein to cause the interaction, the processor is programmed to do so without input from a user (
“For further example, the context parameters 428 may be used to position, orient, and/or scale a selected mixed-reality object or effect in a mixed reality session, based on objects or events sensed in the user’s physical environment. For example, the parameters may be used to cause a mixed-reality object to appear to react to physical objects in the physical environment, such as, for example, to bounce off of or to stick to physical objects. ” Smith ¶ 42.).
Regarding Claim 3, Smith in view of Shapira teaches The system of claim 1, wherein to cause the interaction, the processor is programmed to do so in response to an input from a user (
Smith provides examples input, e.g., gaze, from a user, stating “For example, a mixed reality AR application may detect that the user is viewing a video billboard or other image display appearing in the user’s real environment, and configure the mixed reality content based on the content appearing on the real display in some way. For example, the mixed reality application may be configured to replace the content appearing on a real video display or static image display with a substitute video or static image in the mixed reality session.” Smith ¶ 30. “FIG. 4 shows a system 400 of cooperating components for providing a context-responsive mixed reality output for at least one user 401 who interacts with a mixed reality session 410 operated by at least one computer processor via a user interface 403. ” Smith ¶ 38.),
Regarding Claim 4, Smith in view of Shapira teaches The system of claim 3, wherein the input comprises a user interaction (e.g., through gaze) with the physical object (e.g., billboard/display in real environment) (
Smith provides some examples, stating “For example, a mixed reality AR application may detect that the user is viewing a video billboard or other image display appearing in the user’s real environment, and configure the mixed reality content based on the content appearing on the real display in some way. For example, the mixed reality application may be configured to replace the content appearing on a real video display or static image display with a substitute video or static image in the mixed reality session. In an alternative, or in addition, the mixed reality application may be configured to cause a virtual object or character that relates to content appearing on the real display to appear in the mixed reality session. For example, if the real content includes advertising for a particular product, the application may insert an animated mascot character for the product (or for a competing product) in the mixed reality session. For further example, if the real content depicts a particular scene, object, theme or character, the application may insert virtual content that enhances, parodies, or contradicts the real content.” Smith ¶ 30.).
Regarding Claim 5, Smith in view of Shapira teaches The system of claim 3, wherein the input comprises a user interaction (e.g., through gaze) with the virtual object (e.g., virtual overlay) (Smith provides examples input, e.g., gaze, from a user, stating “For example, a mixed reality AR application may detect that the user is viewing a video billboard or other image display appearing in the user’s real environment, and configure the mixed reality content based on the content appearing on the real display in some way. For example, the mixed reality application may be configured to replace the content appearing on a real video display or static image display with a substitute video or static image in the mixed reality session.” Smith ¶ 30.).
Claims 10-11 and 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Kim (US 20260120311 A1) in view of Neff (US 20140365242 A1).
Regarding Claim 10, Kim teaches A system, comprising: a processor programmed to (Kim fig. 3):
access content comprising text and/or an image (images (110L and 110R) captured by camera, e.g., HMD camera) in a structured or unstructured format (the captured image is either structured or unstructured)(
PNG
media_image2.png
476
714
media_image2.png
Greyscale
“Referring to FIG. 1, the augmented reality device 100 may include a camera 110, and the camera 110 may include a left-eye camera 110L and a right-eye camera 110R. When the user wears the augmented reality device 100, the left-eye camera 110L may be located adjacent to the user's left eye and may photograph the real-world space 10 to obtain a left-eye image.” Shapira ¶ 41.
The term “a structured or unstructured format” appear to include all possibilities.);
identify at least two data objects (Kim Fig. 1 11-15) in the content, each data object representing a virtual object or a real world (RW) object (Kim Fig. 1
PNG
media_image3.png
110
148
media_image3.png
Greyscale
);
learn contextual data (Kim Fig. 1 (2) recognized “Spatial Context”) between the two data objects based at least on the content from which the two data objects were identified (Kim Fig. 1 (2)), the contextual data defining a context in which the two data objects appeared together in the content (Kim Fig. 1 (2));
generate a linked data record (Kim Fig. 1 (2) matched “Spatial Context preset” or Kim ¶ 112’s new/unmatched spatial context) comprising an identification for each of the two data objects (
PNG
media_image4.png
114
246
media_image4.png
Greyscale
) and the learned contextual data (Kim Fig. 1 (2) recognized “Spatial Context”) so that identification of at least one of the data objects is sufficient to identify the linked data record (
[BRI on the record]
With respect to claimed “is sufficient,” the Examiner is reading the limitation to mean “is capable of.”
[Mapping Analysis]
“In operation S850, the augmented reality device 100 may store the new position information. The augmented reality device 100 may store the new spatial context preset in the memory 140. In an embodiment of the present disclosure, the augmented reality device 100 may store the new scene graph as a new spatial context preset in the spatial context database 146 (see FIG. 3).” Kim ¶ 112.
“In an embodiment of the present disclosure, the scene graph may be implemented in the format of extensible markup language (XML) or the like. A node of the scene graph may include a semantic label representing a type or category of a recognized object, and an edge thereof may represent a relative position relationship including at least one of a distance, a direction vector, or a sign vector in a three-dimensional space between a reference object and another object. In an embodiment of the present disclosure, the edge may include not only the relative position relationship between the objects but also information about the dependency relationship between the objects (e.g., monitor and keyboard).” Kim ¶ 86. “FIG. 6 is a diagram illustrating an operation of identifying, by an augmented reality device 100 according to an embodiment of the present disclosure, position relationship information matched to a relative position relationship between a plurality of objects in a real-world space by measuring a similarity of a scene graph 600.” Kim ¶ 98.
All scene graphs with labels are capable of searched and found based on a label.); and
store the linked data record in a database (Kim ¶ 112) to be later retrieved to provide context for one or more of the two data objects (Kim Fig. 1
PNG
media_image4.png
114
246
media_image4.png
Greyscale
spatial context), the database comprising other linked data records of other data objects (Kim Fig. 1
PNG
media_image5.png
134
148
media_image5.png
Greyscale
) (“In operation S850, the augmented reality device 100 may store the new position information. The augmented reality device 100 may store the new spatial context preset in the memory 140. In an embodiment of the present disclosure, the augmented reality device 100 may store the new scene graph as a new spatial context preset in the spatial context database 146 (see FIG. 3).” Kim ¶ 112.),
wherein the stored linked data record represents contextual data learned about the two data objects (Kim Fig. 1
PNG
media_image4.png
114
246
media_image4.png
Greyscale
) and
wherein the linked data record together with the other linked data records represent contextual information of data objects learned from content (
PNG
media_image5.png
134
148
media_image5.png
Greyscale
).
Kim does not explicitly disclose comprising text and/or an image in a structured or unstructured format.
Neff teaches comprising text and/or an image in a structured or unstructured format (“Multiple streams of unstructured or semi-structured input data may be acquired by one or more networked pervasive devices, such as position sensors, measurement devices, audio sensors, video sensors, motion sensors, cameras, wearable sensors with integrated displays, healthcare instruments and so forth. Input data may also be automatically collected by a data miner from one or more external data sources. Such captured information is assimilated by, for example, automatically transforming the unstructured or semi-structured data (e.g., text, audio and/or video stream, images, etc.) into structured data (e.g., patient record). The resulting structured data may be communicated via a network to remotely-located structured data sources.” Neff ¶ 22.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Neff’s unstructured data with Smith. One of ordinary skill in the art would be motivated to support a wide range of data collection devices. Neff ¶ 22.
Regarding Claim 11, Kim further teaches The system of claim 10, wherein the two data objects each comprise text (semantic labels) that represents a respective virtual object or RW object, and wherein to learn contextual data, the processor is programmed to: determine a semantic distance (node edges) between the text representing the respective RW objects to learn a level of similarity between the two data objects (if no edge, when nodes overlap, it represents higher spatial similarity) (
“In an embodiment of the present disclosure, the scene graph may be implemented in the format of extensible markup language (XML) or the like. A node of the scene graph may include a semantic label representing a type or category of a recognized object, and an edge thereof may represent a relative position relationship including at least one of a distance, a direction vector, or a sign vector in a three-dimensional space between a reference object and another object. In an embodiment of the present disclosure, the edge may include not only the relative position relationship between the objects but also information about the dependency relationship between the objects (e.g., monitor and keyboard).” Kim ¶ 86. “FIG. 6 is a diagram illustrating an operation of identifying, by an augmented reality device 100 according to an embodiment of the present disclosure, position relationship information matched to a relative position relationship between a plurality of objects in a real-world space by measuring a similarity of a scene graph 600.” Kim ¶ 98.
The claimed text is mapped to the disclosed text label, which represents objects, virtual or physical, in a scene.).
Regarding Claim 13, Kim further teaches The system of claim 10, wherein to learn contextual data, the processor is programmed to: identify a location associated with either of the two data objects and/or the content from which the two data objects were identified (Kim Fig. 1 (2) recognized “Spatial Context”; Kim ¶ 98.).
Regarding Claim 14, Kim further teaches The system of claim 10, wherein to learn contextual data, the processor is programmed to: identify a time and/or date associated with either of the two data objects and/or the content (scene graph) from which the two data objects were identified (based on measured similarity) (“The augmented reality device 100 may identify a spatial context preset matched to the spatial context among the plurality of spatial context presets (31, 32, 33, . . . ) by identifying a scene graph of which the measured similarity exceeds a preset threshold.” Kim ¶ 49. The content is identified by the scene graph.).
Regarding Claim 15, Kim further teaches The system of claim 10, wherein the processor is programmed to:
receive an event-driven indication (Fig. 8 S850) that new content (new position information according to Fig. 8 S840-850) comprising text and/or an image (labels that describe objects) has been ingested (recognized and incorporated into scene graph) to a data lake (Fig. 1 146:
PNG
media_image5.png
134
148
media_image5.png
Greyscale
) that stores text and/or images in a structured or unstructured format (
“In an embodiment of the present disclosure, the scene graph may be implemented in the format of extensible markup language (XML) or the like. A node of the scene graph may include a semantic label representing a type or category of a recognized object, and an edge thereof may represent a relative position relationship including at least one of a distance, a direction vector, or a sign vector in a three-dimensional space between a reference object and another object. In an embodiment of the present disclosure, the edge may include not only the relative position relationship between the objects but also information about the dependency relationship between the objects (e.g., monitor and keyboard).” Kim ¶ 86. “FIG. 6 is a diagram illustrating an operation of identifying, by an augmented reality device 100 according to an embodiment of the present disclosure, position relationship information matched to a relative position relationship between a plurality of objects in a real-world space by measuring a similarity of a scene graph 600.” Kim ¶ 98.); and
trigger a learning process to learn the contextual data (
PNG
media_image6.png
370
576
media_image6.png
Greyscale
Kim Fig. 9 teaches a further learning process to learn whether the contextual data should be merged with a prestored spatial context preset.).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Neff as applied to Claim 10, in further view of Ando et al. (“Robust Scene Recognition Using Language Models for Scene Contexts”).
Regarding Claim 12, Kim in view of Neff teaches The system of claim 10, wherein the two data objects each comprise images that represents a respective virtual object or RW object (Kim fig. 1)
Kim in view of Neff does not explicitly disclose, but Ando teaches wherein to learn contextual data, the processor is programmed to: determine a number of times that the images are co-located within a same image across a plurality of content (
PNG
media_image7.png
368
518
media_image7.png
Greyscale
X and Y represent images of scene objects co-located in a compound scene. The number of times number of times is determined to estimate collocation frequency.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Ando’s co-location stats with Smith in view of Neff. One of ordinary skill in the art would be motivated to understand/interpret compound/complex scenes. Ando 4.2 Compound scenes.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Liu et al. (US 20210228022 A1) provided examples for claimed “structured” and “unstructured” data:
“ The cooking appliance includes a food support platform configured to support food items (e.g., food support platform 210, such as a rack in an oven or toaster oven, a pan on a stovetop, a plate in a microwave oven, a basket in an air fryer, etc.), one or more first sensors (e.g., one or more first sensors 141) for capturing structured data, including temperature data corresponding to the food items during operation of the cooking appliance (e.g., such as temperature sensors for determining temperature(s) in the cooking chamber or on the cooking surface, and/or inside of the food items), one or more second sensors (e.g., one or more second sensors 142) for capturing unstructured data, including image data corresponding to the food items during the operation of the cooking appliance (e.g., images captured by cameras (e.g., RGB camera(s) and/or infrared camera(s), depth camera(s), etc.) with a field of view directed to food support platform 210), and one or more heating units that are configured to the heat food items placed on the food support platform during the operation of the cooking appliance.” Liu ¶ 123.
However, Liu is not in the same specific technical area of understanding scenes’ context for the generation of augmented reality.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENGXI LIU whose telephone number is (571)270-7509. The examiner can normally be reached M-F 9 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at 571-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHENGXI LIU/Primary Examiner, Art Unit 2611
Prosecution Insights