DETAILED ACTION
This Office Action is in response to the Applicants' communication filed on July 27, 2026, which amends the independent claims 28 and 48-49, add a new claim 50, and presents arguments, is hereby acknowledged. Claims 28-50 are currently pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s arguments filed on July 27, 2026, have been fully considered.
Applicant argues that by this response, Terminal Disclaimer has been filed.
Examiner replies that TD has been filed and approved, ODP rejections to 28 and 48-49 have been withdrawn.
Applicant argues that by this response, the independent claims 28 and 48-49 are hereby amended to add two new limitations “the at least one interconnectable AR object primitive comprising at least one primitive interface through which the at least one interconnectable AR object primitive interacts with other interconnectable AR object primitives” and “wherein the AR sports content set is generated at least in part based on an interaction between the at least two AR object primitives through the at least one primitive interface” in order to overcome the 35 U.S.C. §103 rejection.
Examiner replies that the amended claims with new limitations may overcome the cited portions of the prior arts. However, a newly found art, Carignano, etc. (US 20070038944 A1) teaches that the at least one interconnectable AR object primitive comprising at least one primitive interface through which the at least one interconnectable AR object primitive interacts with other interconnectable AR object primitives (See Carignano: Figs. 1A-D, and [0046], “In operation, the camera 110 creates image data of the environment 120, wherein preferably the image data may correspond to a dynamic state of the environment 120 which may, for instance, be represented by merely moving the camera 110 with respect to the environment 120, or by providing moveable objects within the environment, for instance the marker objects 125, 126, or one or more of the objects 121 . . . 123 may be moveable. For example, the point of view of the environment 120 may be changed by moving around the camera 110 within the environment 120, thereby allowing to observe especially the marker objects 125, 126 from different perspectives so as to enable the assessment of virtual objects created by the generator 150 from different points of view. The image data provided by the camera 110 which may continuously be updated, are received by the identifying means 130, which recognizes the marker objects 125, 126 and enables the tracking of the marker objects 125, 126 once they are identified, even if pattern recognition is hampered by continuously changing the point of view by, for instance, moving the camera 110 or the marker objects 125, 126. For example, after identifying a predefined pattern associated with the marker objects 125, 126 within the image data, the identifying means 130 may inform the combining means 140 about the presence of a marker object within a specified image data area and based on this information, the means 140 may then continuously track the corresponding object represented by the image data used for identifying the marker objects 125, 126, assuming that the marker objects 125, 126 will not vanish over time. In other embodiments, the process of identifying the marker objects 125, 126 may be performed substantially continuously or at least may be repeated on a regular basis so as to confirm the presence of the marker objects 125, 126 and also to verify or enhance the tracking accuracy of the combining means 140. Based on the image data of the environment and the information provided by the identifying means 130, the combining means 140 creates three-dimensional image data and superimposes corresponding three-dimensional image data received from the object generator 150, wherein the three-dimensional object data are permanently updated on the basis of the tracking operation of the means 140. For instance, the means 140 may, based on the information of the identifying means 130, calculate the position of the camera 110 with respect to the marker objects 125, 126 and use this coordinate information for determining the coordinates of a virtual camera, thereby allowing a precise "overlay" of the object data delivered by the generator 150 with the image data of the marker objects 125, 126. The coordinate information also includes data on the relative orientation of the marker objects 125, 126 with respect to the camera 110, thereby enabling the combining means 140 to correctly adapt the orientation of the virtual object. Finally, the combined three-dimensional virtual image data may be presented by the output means 160 in any appropriate form. For example, the output means 160 may comprise appropriate display means so as to visualize the environment 120 including virtual objects associated with the marker objects 125, 126. When operating the system 100 it is advantageous to pre-install recognition criteria for at least one marker object 125, 126 so as to allow a substantially reliable real-time image processing. Moreover, the correlation between a respective marker object and one or more virtual objects may be established prior to the operation of the system 100 or may also be designed so as to allow an interactive definition of an assignment of virtual objects to marker objects. For example, upon user request, virtual objects initially assigned to the marker object 125 may be assigned to the marker object 126 and vice versa. Moreover, a plurality of virtual objects may be assigned to a single marker object and a respective one of the plurality of virtual objects may be selected by the user, by a software application, and the like”; and [0056], “That is, after gathering image data by the optical imaging system 310, a marker object is identified, for instance attached to a user positioned so as to face the optical imaging system 310 and the display surface 360, and the marker object is identified and tracked, as is previously described. Moreover, based on a predefined or user-initiated association of one or more marker objects with respective virtual objects, the generator 350 provides corresponding object data to the combining means 340, which may produce corresponding virtual image data including the virtual object”. Note that the marker object and the virtual object is predefined or user-initiated, and this is mapped to the interconnectable AR object primitive).
Examiner further replies that Gagner, etc. (US 20110065496 A1) teaches that wherein the AR sports content set is generated at least in part based on an interaction between the at least two AR object primitives through the at least one primitive interface (See Gagner: Figs. 1-2 and [0021], “In one implementation, at stage A, the gaming machine 160 detects a fiducial marker in one or more images captured by a camera 165 of the gaming machine 160. In one example, a player positions a game ticket 122 (or other object) including a fiducial marker 124 in the field of vision of the camera 165. The camera 165 captures video of objects in the field of vision of the camera 165 including the game ticket 122 with the fiducial marker 124. In this example, an image processing mechanism of the gaming machine 160 detects the fiducial marker 124 on the game ticket 122 captured in the video. In one implementation, the fiducial marker 124 can include a fiducial code 125 and a bounding indicator 126. In some examples, the fiducial code 125 may include a 1D barcode, 2D barcode, geometric patterns, text, or a combination thereof, that can be used to identify which augmented reality object to render. The fiducial code 125 may also include additional metadata and other information associated with the fiducial marker, e.g., a serial number to track when the player uses the fiducial marker. In one example, the bounding indicator 126 may include a bounding square or similar indicator that surrounds the fiducial code 125 and helps to identify the location of the fiducial code 125. The bounding indicator 126 can also indicate the orientation of the fiducial marker 124, as will be further described below”; [0027], “At stage F, the wagering game server 150 determines how to render the augmented reality object based on the fiducial code 125 and the orientation of the fiducial marker 124. For example, metadata within the fiducial code 125 and/or the orientation of the fiducial marker 124 can determine attributes of an augmented reality 3D object, e.g., determine movement, orientation, composition, etc. of the augmented reality 3D object”; [0028], “At stage G, the wagering game server 150 composites the augmented reality object with the wagering game content that is being provided to the gaming machine 160. In some instances, compositing can include combining visual elements from separate sources into single images, e.g., one or more images that comprise a video stream. In one implementation, the wagering game server 150 composites video of the fiducial marker 124 and the augmented reality object with the wagering game content. In some implementations, the wagering game server 150 uses the fiducial marker 124 as a reference point when performing the compositing operations. It is noted, however, that in other implementations the gaming machine 160 can be configured to perform the compositing operations. In these implementations, the wagering game server 150 can provide the wagering game content and the rendered augmented reality object to the gaming machine 160, and the gaming machine 160 can composite the video of the fiducial marker 124 and the augmented reality object with the wagering game content”; and [0031], “FIG. 2 is a conceptual diagram illustrating an example of using the fiducial marker 124 to control various attributes of an augmented reality 3D object, according to some embodiments. In the example shown in FIG. 2, during a wagering game session, the gaming machine 160 presents on the display device 166 wagering game content composited with video of the fiducial marker 124 and the associated augmented reality 3D object. In one implementation, the player can rotate, tilt, or otherwise change the orientation of the fiducial mark 124 (while the fiducial marker 124 is positioned in the field of vision of the camera 165) to control various attributes of the augmented reality 3D object during game play. For example, modifying the orientation of the fiducial marker 124 can control the movement, modify the orientation, change the composition, etc. of the augmented reality 3D object, and allow the player to interact with and play the wagering game. In the example of FIG. 2, the wagering game may be a secondary bonus game (e.g., a picking game) and the augmented reality 3D object is an avatar 175 holding a flashlight. In this example, modifying the orientation of the fiducial marker 124 can control the direction the avatar 175 walks within the game, control the direction the avatar 175 points the flashlight, change the orientation of the avatar 175 and flashlight with respect to the point of view of the player, control what other tools (besides the flashlight) the avatar 175 uses, controls other movements of the avatar 175 (e.g., opening doors), etc. It is noted that in other examples various types of augmented reality objects can be used to play various types of wagering games in a similar manner”. Note that detecting the fiducial markers, rendering the specific AR objects based on the marker identity, compositing the AR objects into the scene, and controlling attributes of the AR object is mapped to the AR sports content set generated based on the interaction between the AR objects primitives associated with markers through the primitive interface). The teachings of Carignano and Gagner provide interactive, rule-driven behaviors of marker-associated interconnectable AR objects and generating AR content set based on the interactions between the AR object. Therefore, the remaining arguments of the applicant are mooted in view of these newly found arts.
Examiner respectfully further replies that the Applicant's arguments have been fully considered and a new ground of rejections have been made. Accordingly, new grounds of rejection are set forth below. Since the new grounds of rejection are necessitated by Applicant's amendments to the claims, the present action is made final.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 28-32, 34-38, and 40-49 are rejected under 35 U.S.C. 103 as being unpatentable over Muthyala, etc. (US 20140378023 Al), in view of Dooley, etc. (US 20110270135 Al), further in view of Carignano, etc. (US 20070038944 A1) and Gagner, etc. (US 20110065496 A1).
Regarding claim 28, Muthyala teaches that an augmented reality (AR) sport event system (See Muthyala: Fig. 2, and [0050], "FIG. 2 shows an embodiment of a toy construction system as disclosed herein. The system comprises a computer 15, a display 1, a video camera 5, a toy construction model 10 constructed from at least one toy construction element and at least one marker construction element 11 of a toy construction system as described herein. The computer may be a personal computer, a desktop computer, a laptop computer, a handheld computer, a game console, a handheld entertainment device, or any other suitably programmable computer. The display 1 is operatively coupled to (e.g. integrated into) the computer 15, and operable to display, under the control of the computer 15, a video image. It will be appreciated that the system may comprise further peripheral computer devices operatively coupled to the computer 15, such as a keyboard, a mouse 2 or other pointing device, and/or the like. The video camera 5 is operable to capture video images of the environment in which the video camera is situated, e.g. of a field of view 7 including a visible surface area 8 of a surface 4, e.g. a desktop, a floor, or the like. Hence the visible surface area 8 is the projection of the field of view of the video camera on the surface 4. The video camera is operatively coupled to (e.g. integrated into) the computer 15 and operable to forward the captured video image to a processing unit of the computer 15, e.g. via a suitable input interface of the computer 15. For example, the video camera may be a webcam connected to or integrated into computer 15. In the example of FIG. 2, the video camera is positioned on a camera support 6, such as a tripod, supporting the camera at a predetermined height above the surface 4"), comprising:
at least one non-transitory computer readable memory storing software instructions (See Muthyala: Fig. 2, and [0032], "The computer program product may be provided as a computer-readable medium, such as a CD-ROM, DVD, optical disc, memory card, flash memory, magnetic storage device, floppy disk, hard disk, etc. In other embodiments, a computer program product may be provided as a downloadable software package, e.g. on a web server for download over the Internet or other computer or communication network. In particular, an embodiment of a toy construction set may include toy construction elements including one or more marker construction elements as described herein, and installation instructions for installing a computer program product on a suitable data processing system"); and
at least one processor that, upon execution of the software instructions, performs operations (See Muthyala: Fig. 2, and [0031], "In particular, a method is provided for generating an image of a toy construction model. Furthermore, a computer program product is provided comprising program code means adapted to cause, when executed on a data processing system, a data processing system to perform the steps of the method described herein") comprising:
receiving a digital representation of an arrangement of physical markers in a real-world sports environment (See Muthyala: Figs. 1-3, and [0056], "The computer 15 is operable to detect the presence of the two-dimensional machine readable code in the captured image. Furthermore, the computer 15 may determine a relative position and orientation of the two-dimensional machine readable code relative to the position of the video camera 15"; [0057], "Accordingly, the computer 15 may modify the captured image of the toy construction model resulting in a modified video image 12 displayed on the display 1. In this example, the modified image shows the captured image 13 of the toy construction model and computer-generated image elements 14 superimposed the captured image. The computer generates the computer-generated image elements at a position and orientation within the video image 12 as determined from the detected position and orientation of the AR marker of the marker construction element 11"; and [0063], "The AR marker may be arranged in a uniform manner relative to the coupling means, i.e. to the coupling studs on the top surface and/or to the coupling cavity in the bottom. For example, the AR marker may define a direction parallel or normal to the planar grid defined by the coupling means. This makes the marker bricks interchangeable, and in a toy structure built from bricks as in FIGS. 1-3, several marker bricks can be used interchangeably, and a particular marker brick can be used in several constructions. A toy construction system may comprise several of such marker bricks having different insignia applied to it and causing the computer system to generate different computer-generated images. Nevertheless, if all marker bricks include the insignia at uniform positions, such marker bricks may easily be interchanged within a toy construction built from the building bricks described herein". Note that the 2D machine readable code is mapped to the digital representation);
identifying each marker's identity from digital features derived from the digital representation (See Muthyala: Figs. 5A-G, and [0067], "FIGS. 5a-g illustrate examples of composite marker construction elements, i.e. a plurality of marker construction elements directly or indirectly connected with each other by means of the coupling means of the toy construction system, e.g. in a predetermined spatial relationship with each other. Generally, composite marker construction elements allow the user to create a large set of distinct objects, each identifiable by one of a large number of unique AR markers. In particular, the large number of unique AR markers may be created from a relatively limited set of individual marker construction elements, because combining a smaller set of markers through a building system dramatically increases the total number of combinatorial possibilities");
identifying at least one interconnectable AR object primitive for each identified marker (See Muthyala: Fig. 1-5, and [0065], "FIG. 4 illustrates how a marker construction element may be coupled to another toy construction element of a toy construction model. In this example, the toy construction model comprises toy construction elements 410 and 412 and a marker construction element 411. FIG. 4a shows the individual constructions elements, while FIG. 4b shows the marker construction element 411 releasably coupled to the construction element 410. To this end, the toy construction element 410 comprises coupling studs 405 on its top surface as described in connection with FIG. 1. The marker construction element 404 comprises cavities on its bottom surface (not shown in FIG. 4), allowing it to frictionally engage the coupling studs 404"); and [0066], "Hence, the user may connect a marker construction element comprising an AR marker to a constructed toy model so as to facilitate tracking the toy model in 3D space and to superimpose virtual graphics and effects, thereby providing the physical toy with virtual attributes and allow the constructed toy model to become a part of an AR application. As in the example of FIG. 3, the insignia of the marker construction element 411 is located on a surface of the marker construction element that does not comprise coupling element". Note that the physical pieces with markers is a specific virtual model, and each marker (on the physical blocks, construction elements) maps to a virtual 3D object or "primitive", and the physical/virtual coupling rules are mapped to "interconnectable"), the at least one interconnectable AR object primitive comprising at least one primitive interface through which the at least one interconnectable AR object primitive interacts with other interconnectable AR object primitives;
generating an AR sports content set by coupling at least two AR object primitives according to a sports application-specific rules set and the arrangement of the physical markers (See Muthyala: Figs. 9A-B, and [0090], "FIG. 9 shows toy construction elements 911a-911b of the type shown in FIG. 1. In this embodiment, the different construction elements have respective insignia, different color, and/or another distinguishable visual feature. Each of the toy construction element, when detected by the AR system within a toy construction model, triggers a predetermined action. For example, in the example of FIG. 9, each of the construction elements 911a-911d represents a different type of landscape or environmental phenomena, such as gras, water, snow, fire, etc. When detected by the AR system, each construction element may thus be augmented by the AR system by displaying corresponding image elements representing grass, water, fire, snow, respectively. This allows a user to build a landscape by means of the construction elements, e.g. as illustrated in gig. 9b"), wherein the AR sports content set is generated at least in part based on an interaction between the at least two AR object primitives through the at least one primitive interface; and
causing a device to present the AR sports content set superimposed on an image of the real-world sports environment (See Muthyala: Fig. 9A-B, and [0006], "Within the different technical field of image recognition and computer graphics, systems that utilize augmented reality (AR) have attracted increased interest. Augmented reality (AR) is a technology where a captured live view of one or more items of a physical, real-world environment is augmented by a computer-generated graphics, i.e. where a user is presented with a composite view comprising the live view and the computer-generated graphics, e.g. in the form of an overlay/superposition of the live view with the computer-graphics"; and [0091], "It will be appreciated that the marker construction elements may also trigger further actions within the AR system. For example, the landscape of FIG. 9b may be used in the AR system as a landscape within which a game is performed, i.e. the user may build his/her own levels of a computer game by using physical building elements").
However, Muthyala fails to explicitly disclose that the at least one interconnectable AR object primitive comprising at least one primitive interface through which the at least one interconnectable AR object primitive interacts with other interconnectable AR object primitives; generating an AR sports content set; and wherein the AR sports content set is generated at least in part based on an interaction between the at least two AR object primitives through the at least one primitive interface.
However, Dooley teaches that generating an AR sports content set (See Dooley: Fig. 1-7, and [0094], "Referring to FIG. 7, the wearable display 220 depicts real world images seen through the glasses 220 that include three trees, and virtual reality cues overlaid on the real world images. The virtual reality depicts a start and a racing hurdle on the right glass and an arrow on the left glass. The arrow tells the user that she must jump higher to clear the hurdle. Although the right and left glasses show different images, the user sees the three trees, hurdle and arrow as a single display"; and [0096], "In step 320, the personal computer 111 reads the three dimensional information from the source 110 and uses transmitter 112 to transmit the information wirelessly to receiver 103. This step is necessary because the active controller 101 transmits the data directly to the source unit 110. If the transmission protocol were known and was able to be mimicked by the body worn computing device 102, this step would not be needed, as the computing device 102 could simply communicate with the active controller 101 directly. In step 325, the computing device 102 generates the virtual simulation using the positional and orientation data from the passive controllers l00A-F and displays the information on the wearable display 120". Note that the generated AR sports content is common application of AR system in virtual game field).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Muthyala to have generating an AR sports content set as taught by Dooley in order to identify kinematic and/or performance factors that may expose the athlete to an increased risk of injury or that negatively impact the athlete's physical performance capabilities (See Dooley: Fig. 1, and [0062], "The present invention is capable of identifying kinematic and/or performance factors that may expose the athlete to an increased risk of injury or that negatively impact the athlete's physical performance capabilities. Real-time visual feedback can alert the athlete to potentially dangerous, or at least potentially inefficient, movement patterns. Alternatively, the present invention can be used as an entertaining physical activity for members of the general populous, including children, seniors, patients and fitness buffs"). Muthyala teaches a method and system that may generate AR content using the physical markers arranged in the real word environment to identify and configure marker based interconnectable virtual primitives (building blocks, scene, trees) with rules sets for virtual primitive arrangements; while Dooley teaches a system and method that may generate AR content in sports environment by applying sports rules of valid field layout and event element. Therefore, it is obvious to one of ordinary skill in the art to modify Muthyala by Dooley to substitute sports rules, layouts, event elements for the arrangement rules, virtual primitives, etc. to generate AR sports content set. The motivation to modify Muthyala by Dooley is "Simple substitution of one known element for another to obtain predictable results".
However, Muthyala, modified by Dooley, fails to explicitly disclose that the at least one interconnectable AR object primitive comprising at least one primitive interface through which the at least one interconnectable AR object primitive interacts with other interconnectable AR object primitives; and wherein the AR sports content set is generated at least in part based on an interaction between the at least two AR object primitives through the at least one primitive interface.
However, Carignano teaches that the at least one interconnectable AR object primitive comprising at least one primitive interface through which the at least one interconnectable AR object primitive interacts with other interconnectable AR object primitives (See Carignano: Figs. 1A-D, and [0046], “In operation, the camera 110 creates image data of the environment 120, wherein preferably the image data may correspond to a dynamic state of the environment 120 which may, for instance, be represented by merely moving the camera 110 with respect to the environment 120, or by providing moveable objects within the environment, for instance the marker objects 125, 126, or one or more of the objects 121 . . . 123 may be moveable. For example, the point of view of the environment 120 may be changed by moving around the camera 110 within the environment 120, thereby allowing to observe especially the marker objects 125, 126 from different perspectives so as to enable the assessment of virtual objects created by the generator 150 from different points of view. The image data provided by the camera 110 which may continuously be updated, are received by the identifying means 130, which recognizes the marker objects 125, 126 and enables the tracking of the marker objects 125, 126 once they are identified, even if pattern recognition is hampered by continuously changing the point of view by, for instance, moving the camera 110 or the marker objects 125, 126. For example, after identifying a predefined pattern associated with the marker objects 125, 126 within the image data, the identifying means 130 may inform the combining means 140 about the presence of a marker object within a specified image data area and based on this information, the means 140 may then continuously track the corresponding object represented by the image data used for identifying the marker objects 125, 126, assuming that the marker objects 125, 126 will not vanish over time. In other embodiments, the process of identifying the marker objects 125, 126 may be performed substantially continuously or at least may be repeated on a regular basis so as to confirm the presence of the marker objects 125, 126 and also to verify or enhance the tracking accuracy of the combining means 140. Based on the image data of the environment and the information provided by the identifying means 130, the combining means 140 creates three-dimensional image data and superimposes corresponding three-dimensional image data received from the object generator 150, wherein the three-dimensional object data are permanently updated on the basis of the tracking operation of the means 140. For instance, the means 140 may, based on the information of the identifying means 130, calculate the position of the camera 110 with respect to the marker objects 125, 126 and use this coordinate information for determining the coordinates of a virtual camera, thereby allowing a precise "overlay" of the object data delivered by the generator 150 with the image data of the marker objects 125, 126. The coordinate information also includes data on the relative orientation of the marker objects 125, 126 with respect to the camera 110, thereby enabling the combining means 140 to correctly adapt the orientation of the virtual object. Finally, the combined three-dimensional virtual image data may be presented by the output means 160 in any appropriate form. For example, the output means 160 may comprise appropriate display means so as to visualize the environment 120 including virtual objects associated with the marker objects 125, 126. When operating the system 100 it is advantageous to pre-install recognition criteria for at least one marker object 125, 126 so as to allow a substantially reliable real-time image processing. Moreover, the correlation between a respective marker object and one or more virtual objects may be established prior to the operation of the system 100 or may also be designed so as to allow an interactive definition of an assignment of virtual objects to marker objects. For example, upon user request, virtual objects initially assigned to the marker object 125 may be assigned to the marker object 126 and vice versa. Moreover, a plurality of virtual objects may be assigned to a single marker object and a respective one of the plurality of virtual objects may be selected by the user, by a software application, and the like”; and [0056], “That is, after gathering image data by the optical imaging system 310, a marker object is identified, for instance attached to a user positioned so as to face the optical imaging system 310 and the display surface 360, and the marker object is identified and tracked, as is previously described. Moreover, based on a predefined or user-initiated association of one or more marker objects with respective virtual objects, the generator 350 provides corresponding object data to the combining means 340, which may produce corresponding virtual image data including the virtual object”. Note that the marker object and the virtual object is predefined or user-initiated, and this is mapped to the interconnectable AR object primitive).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Muthyala to have the at least one interconnectable AR object primitive comprising at least one primitive interface through which the at least one interconnectable AR object primitive interacts with other interconnectable AR object primitives as taught by Carignano in order to allow enhanced flexibility and improved fidelity in tracking and superimposing virtual objects with a specified real environment (See Carignano: Fig. 1, and [0003], "Therefore, a need exists for an augmented reality system and a method that allows enhanced flexibility and improved "fidelity" in tracking and superimposing virtual objects with a specified real environment "). Muthyala teaches a method and system that may generate AR content using the physical markers arranged in the real word environment to identify and configure marker based interconnectable virtual primitives (building blocks, scene, trees) with rules sets for virtual primitive arrangements; while Carignano teaches a system and method that may gather image data of a real environment, generate virtual image data from said image data, identify a predefined marker object of the real environment based on the image data, and superimpose a set of object image data with the virtual image data at a virtual image position corresponding to the predefined marker object by associating the virtual object with the markers based on user request. Therefore, it is obvious to one of ordinary skill in the art to modify Muthyala by Carignano to associate the virtual objects with markers dynamically to improve tracking accuracy. The motivation to modify Muthyala by Carignano is " Use of known technique to improve similar devices (methods, or products) in the same way ".
However, Muthyala, modified by Dooley and Carignano, fails to explicitly disclose that wherein the AR sports content set is generated at least in part based on an interaction between the at least two AR object primitives through the at least one primitive interface.
However, Gagner teaches that wherein the AR sports content set is generated at least in part based on an interaction between the at least two AR object primitives through the at least one primitive interface (See Gagner: Figs. 1-2 and [0021], “In one implementation, at stage A, the gaming machine 160 detects a fiducial marker in one or more images captured by a camera 165 of the gaming machine 160. In one example, a player positions a game ticket 122 (or other object) including a fiducial marker 124 in the field of vision of the camera 165. The camera 165 captures video of objects in the field of vision of the camera 165 including the game ticket 122 with the fiducial marker 124. In this example, an image processing mechanism of the gaming machine 160 detects the fiducial marker 124 on the game ticket 122 captured in the video. In one implementation, the fiducial marker 124 can include a fiducial code 125 and a bounding indicator 126. In some examples, the fiducial code 125 may include a 1D barcode, 2D barcode, geometric patterns, text, or a combination thereof, that can be used to identify which augmented reality object to render. The fiducial code 125 may also include additional metadata and other information associated with the fiducial marker, e.g., a serial number to track when the player uses the fiducial marker. In one example, the bounding indicator 126 may include a bounding square or similar indicator that surrounds the fiducial code 125 and helps to identify the location of the fiducial code 125. The bounding indicator 126 can also indicate the orientation of the fiducial marker 124, as will be further described below”; [0027], “At stage F, the wagering game server 150 determines how to render the augmented reality object based on the fiducial code 125 and the orientation of the fiducial marker 124. For example, metadata within the fiducial code 125 and/or the orientation of the fiducial marker 124 can determine attributes of an augmented reality 3D object, e.g., determine movement, orientation, composition, etc. of the augmented reality 3D object”; [0028], “At stage G, the wagering game server 150 composites the augmented reality object with the wagering game content that is being provided to the gaming machine 160. In some instances, compositing can include combining visual elements from separate sources into single images, e.g., one or more images that comprise a video stream. In one implementation, the wagering game server 150 composites video of the fiducial marker 124 and the augmented reality object with the wagering game content. In some implementations, the wagering game server 150 uses the fiducial marker 124 as a reference point when performing the compositing operations. It is noted, however, that in other implementations the gaming machine 160 can be configured to perform the compositing operations. In these implementations, the wagering game server 150 can provide the wagering game content and the rendered augmented reality object to the gaming machine 160, and the gaming machine 160 can composite the video of the fiducial marker 124 and the augmented reality object with the wagering game content”; and [0031], “FIG. 2 is a conceptual diagram illustrating an example of using the fiducial marker 124 to control various attributes of an augmented reality 3D object, according to some embodiments. In the example shown in FIG. 2, during a wagering game session, the gaming machine 160 presents on the display device 166 wagering game content composited with video of the fiducial marker 124 and the associated augmented reality 3D object. In one implementation, the player can rotate, tilt, or otherwise change the orientation of the fiducial mark 124 (while the fiducial marker 124 is positioned in the field of vision of the camera 165) to control various attributes of the augmented reality 3D object during game play. For example, modifying the orientation of the fiducial marker 124 can control the movement, modify the orientation, change the composition, etc. of the augmented reality 3D object, and allow the player to interact with and play the wagering game. In the example of FIG. 2, the wagering game may be a secondary bonus game (e.g., a picking game) and the augmented reality 3D object is an avatar 175 holding a flashlight. In this example, modifying the orientation of the fiducial marker 124 can control the direction the avatar 175 walks within the game, control the direction the avatar 175 points the flashlight, change the orientation of the avatar 175 and flashlight with respect to the point of view of the player, control what other tools (besides the flashlight) the avatar 175 uses, controls other movements of the avatar 175 (e.g., opening doors), etc. It is noted that in other examples various types of augmented reality objects can be used to play various types of wagering games in a similar manner”. Note that detecting the fiducial markers, rendering the specific AR objects based on the marker identity, compositing the AR objects into the scene, and controlling attributes of the AR object is mapped to the AR sports content set generated based on the interaction between the AR objects primitives associated with markers through the primitive interface. The teachings of Carignano and Gagner provide interactive, rule-driven behaviors of marker-associated interconnectable AR objects and generating AR content set based on the interactions between the AR object).
Therefore, it would have been obvious to one of ordinary skill in the art before wherein the AR sports content set is generated at least in part based on an interaction between the at least two AR object primitives through the at least one primitive interface as taught by Gagner in order to continuously develop new games and gaming enhancements that attract frequent play (See Gagner: Fig. 1, and [0003], " Shrewd operators consequently strive to employ the most entertaining and exciting machines, features, and enhancements available because such machines attract frequent play and hence increase profitability to the operator. Therefore, there is a continuing need for wagering game machine manufacturers to continuously develop new games and gaming enhancements that will attract frequent play "). Muthyala teaches a method and system that may generate AR content using the physical markers arranged in the real word environment to identify and configure marker based interconnectable virtual primitives (building blocks, scene, trees) with rules sets for virtual primitive arrangements; while Gagner teaches a system and method that may detect a fiducial marker in images captured by an image capture device of a gaming machine, identify an augmented reality object with the code and control attributes of the object through associating the virtual objects with the marker. Therefore, it is obvious to one of ordinary skill in the art to modify Muthyala by Gagner to associate the virtual objects with the markers and control the virtual object attributes through the associated markers interactively. The motivation to modify Muthyala by Gagner is " Use of known technique to improve similar devices (methods, or products) in the same way ".
Regarding claim 29, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Dooley teaches that the system of claim 28, wherein the physical markers comprise physical equipment in the real-world sports environment (See Dooley: Fig. 5, and [0084], "Without proper registration of the digital information, the ability of the system to measure the physical performance or kinematics of the user, or for the static and dynamic objects to realistically interact with the user may be dampened. Distinguishable objects ("markers") placed in the physical space may play an important role to AR's performance. US 2004/0080548 teaches the use "of a plurality of at least three tracking fiducials selectively each respectively located in fixed predetermined locations in the observation space ... "To effectively enable the present invention combined with AR, proper means to register and precisely align the real and virtual views is advantageous". Note that the markers are attached or placed in the real world environment, and linked to the real or virtual objects which are mapped to the physical equipment (objects)).
Regarding claim 30, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Dooley teaches that the system of claim 28, wherein the AR sports content set comprises virtual spectators (See Dooley: Fig. 7, and [0060], "FIG. 7 depicts an optical overlay-based augmented reality system. Depicted here is a cluster of three trees in a real world landscape. The viewer sees the landscape as a unit when she looks through the glasses with both eyes". Note that the viewers are mapped to the spectators).
Regarding claim 31, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Muthyala teaches that the system of claim 28, wherein the operations further comprise updating the AR sports content set in real-time based on movement of the physical markers (See Muthyala: Fig. 11, and [0098], "For example, when in the closed position, the AR system may augment a live view of a toy structure comprising the door with computer-generated images of movable objects (e.g. human- or animal like figures) that move around the environment defined by the toy structure. When the AR system detects the door to be closed, the movement of the movable objects may constrained to only a portion of the environment (or no objects may be visible at all), while the movable objects may be shown to pass through an opening covered by the door, when the door is detected to be in the open position"; and [0008], "To this end the box comprises an augmented reality marker on its outside. When the user holds the box in front of a video camera of an in-store kiosk system, the kiosk system shows the captured video image of the box, augmented with a computer-generated image of the construction model that is shown as if built on top of the box. When the user moves or rotates the box in front of the camera, the computer-generated image of the construction model follows these movements").
Regarding claim 32, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Muthyala teaches that the system of claim 28, wherein the application-specific rules set is selected based on contextual information associated with the real-world sports environment (See Muthyala: Fig. 9, and [0090], "FIG. 9 shows toy construction elements 911a-911b of the type shown in FIG. 1. In this embodiment, the different construction elements have respective insignia, different color, and/or another distinguishable visual feature. Each of the toy construction element, when detected by the AR system within a toy construction model, triggers a predetermined action. For example, in the example of FIG. 9, each of the construction elements 911a-911d represents a different type of landscape or environmental phenomena, such as gras, water, snow, fire, etc. When detected by the AR system, each construction element may thus be augmented by the AR system by displaying corresponding image elements representing grass, water, fire, snow, respectively. This allows a user to build a landscape by means of the construction elements, e.g. as illustrated in gig. 9b". Note that the different markers representing different primitives (water, snow, fires, etc.) may be mapped to the contextual information).
Regarding claim 34, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Muthyala teaches that the system of claim 28, wherein the AR sports content set includes at least one virtual obstacle superimposed on the real-world sports environment (See Muthyala: Fig. 11, and [0096], "FIGS. 11 a-b illustrate an example of a marker construction element comprising two selectively visible AR markers. FIGS. 11 a-b schematically show a toy construction model 1101 comprising a door 1103 hinged to a frame 1102 to allow the door 1103 to pivot between a closed position as shown in FIG. 11a and an open position as shown in FIG. 11 b. The frame and the door may be releasably interconnectable with each other, e.g. by snapping the hinges into place. Alternatively or additionally, the frame 1102 may comprise one or more coupling elements (not explicitly shown in FIG. 11) allowing the frame and door structure to be connected to a larger toy construction model, e.g. so as to provide a movable barrier covering an opening in a toy structure").
Regarding claim 35, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Dooley teaches that the system of claim 28, wherein the operations further comprise generating a personalized AR interface based on locations in the real-world sports environment (See Dooley: Fig. 7, and [0036], "These aforementioned objectives can be achieved by the use of augmented reality ("AR"). The present invention's use of AR enables the delivery of essentially continuous realtime visual feedback relating to the athlete's physical performance and/or the athlete's kinematics regardless of the vector direction in which the athlete is transiting or the direction to which the athlete is gazing (her viewpoint). With the present invention, the athlete wears a suitable Head-Mounted-Display ("HMD"); examples of suitable HMDs include optical see-through and video see-through HMDs. HMDs can also be referred to as "wearable displays." Simply stated, AR augments reality. It superimposes digital information on top of the athlete's real world (natural) view of his/her surrounding environment. AR may also add sound and haptics to the real world view. Noted AR researcher Ron Azuma defines AR as "a technology which: (1) combines real and virtual imagery, (2) is interactive in real time, (3) registers the virtual imagery with the real world." Unlike the previously discussed sports simulators and virtual reality, AR provides both a real-world view and a view of overlaid computer-generated graphics. This graphical overlay serves to provide visual stimuli (cuing) and visual feedback relating to the athlete's physical performance and the athlete's kinematics (form) during locomotion"; and [0038], "The graphical overlay could take many forms. For example, static virtual object(s) could be "placed" in the real-world view at perceived locations and distances replicating a traditional cone drill, such as is frequently used to test the agility of athletes. Upon viewing the virtual cones, the athlete could initiate movement within the real world physical space to that perceived physical location where a virtual (graphic) cone or cones have been overlaid on the real-world view").
Regarding claim 36, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Muthyala teaches that the system of claim 28, wherein the physical markers include one or more markers selected from the group consisting of: a card associated with the real-world sports environment, a trading card associated with the real-world sports environment, a toy associated with the real-world sports environment, and an action figure associated with the real-world sports environment (See Muthyala: Figs. SA-G, and [0072], "As illustrated by FIG. Sg, the insignia of the marker construction elements may define an orientation in a plane, and individual marker elements may be rotated with respect to each other, thus further increasing the degrees of freedom of defining multiple codes, as the AR system may detect the orientation of the respective AR markers relative to each other and/or relative to another feature of the toy construction model"; and [0061], "It will be appreciated that other forms of AR markers may be used instead of a 2D bar code. Generally, the AR marker may be any object that is relatively easy to automatically distinguish within the captured video images using known image recognition methods. Where the AR marker is detected using image recognition methods, the AR marker may be, for example, a three dimensional object such as a cube or a cylinder or it may be a two dimensional marker such as a square or a circle. Typically, the AR marker comprises an easily distinguishable pattern such as a black and white square although other methods may be used for marker recognition such as using a particular colour or pattern of colours and the like").
Regarding claim 37, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Muthyala teaches that the system of claim 28, wherein the operations further comprise modifying the AR sports content set based on sensor data from the environment (See Muthyala: Fig. 2, and [0003], "It is generally desirable to enhance the play value of physical construction models and to allow users to interact with physical construction models via a computer. To this end, physical construction models have been disclosed that include programmable processors that accept input from sensors and can activate function elements in response to received sensor inputs, and that may be programmed by the user by means of a suitably programmed computer").
Regarding claim 38, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Dooley teaches that the system of claim 28, wherein the operations further comprise enabling interactions with the AR sports content set through a mobile device (See Dooley: Figs. 3-4, and [0101], "In step 430 the computing device 202 generates the virtual simulation using the positional and orientation data from the sensors 200A-E and displays the information on the wearable display 220. The wearable display is preferably an optical see-through HMD from Microvision, but at the current time no model is available to the public. Instead, a video see-through HMD from Vuzix (e.g. WRAP 920AR+) is employed. Since the display obscures the user's vision, the 920AR+ contains two video cameras that record the user's natural world (his/her viewpoint). Since the wearable display 220 cannot overlay the simulation directly onto the screen, there is an extra step the computing device 202 needs to perform. The computing device 202 needs to take the video obtained from the integrated video cameras in the wearable display and combine those images with the simulation currently in progress. This combined picture of the real (natural) world plus the simulation (virtual) world can then be displayed to the user on the wearable display. This step would not be necessary with optical see-through displays. In an optical see-through display the wearable display is transparent and the simulation can be projected directly onto the screen and the user can see the natural world behind the display". Note that the HMD with wearable display is mapped to the mobile device).
Regarding claim 40, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Dooley teaches that thee system of claim 28, wherein the operations further comprise generating a simulation including a digital model associated with the real-world sports environment (See Dooley: Figs. 2-6, and [0094], "Referring now to FIGS. 3, 4 and 6, an alternative embodiment of the present invention is depicted that includes a source 203 that is body worn and generates a magnetic field which is detected by the passive controllers 200A-E. The passive controllers 200A-E communicate with an active controller 201 can via wired or wireless transmission. The active controller 201 then communicates the position and orientation of all of the passive controllers 200A-E back to the source 203 via wireless transmission. A body worn computing device 202 (e.g., a personal computer, smart phone, iPod, or other computing system) is connected to the source 203 and communicates with the source 202 via wired or wireless transmission (e.g. Bluetooth, RF, etc.). The computing device 202 is also coupled to a GPS receiver 204A or other means for determining the exact position in free space (e.g. RFID Tags, Indoor GPS, etc.) and also a 6-axis sensor 204B, which contains a 3-axis accelerometer and a 3-axis gyroscope. The computing device 202 processes the received data from all three sources 203, 204A and 204B and integrates the data into the running simulation. The computing device 202 is coupled via cable, or other means to a wearable display 220 for display output of the simulation in operation that includes continuously providing realtime visual physical performance information to the user while the user is moving to enable the user to detect physical performance constructs that expose the user to increased risk of injury or that reduce the user's physical performance. Referring to FIG. 7, the wearable display 220 depicts real world images seen through the glasses 220 that include three trees, and virtual reality cues overlaid on the real world images. The virtual reality depicts a start and a racing hurdle on the right glass and an arrow on the left glass. The arrow tells the user that she must jump higher to clear the hurdle. Although the right and left glasses show different images, the user sees the three trees, hurdle and arrow as a single display". Note that the running simulation model is mapped to the digital model).
Regarding claim 41, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Muthyala teaches that the system of claim 28, wherein the AR sports content set is based on a juxtaposition of the physical markers (See Muthyala: Fig. 2, and [0057], "Accordingly, the computer 15 may modify the captured image of the toy construction model resulting in a modified video image 12 displayed on the display 1. In this example, the modified image shows the captured image 13 of the toy construction model and computer-generated image elements 14 superimposed the captured image. The computer generates the computer-generated image elements at a position and orientation within the video image 12 as determined from the detected position and orientation of the AR marker of the marker construction element 11". Note that the relative marker positions and orientations are mapped to the juxtaposition of markers).
Regarding claim 42, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Muthyala teaches that the system of claim 28, wherein the operations further comprise tracking position of the physical markers in real-time when generating the AR sports content set (See Muthyala: Fig. 2, and [0058], "As the user manipulates the physical toy construction model 10 within the projection area 8 of the video camera, e.g. by moving and/or rotating the physical model, the computer 15 tracks the position and orientation of the insignia of the marker construction element 11 of the physical toy. The computer 15 displays the live video feed of the video camera (mirrored mode) on the display 1 and adds, responsive to the detected position and orientation of the insignia, augmented reality special effects to the live video feed").
Regarding claim 43, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Dooley teaches that the system of claim 28, wherein the AR sports content set represents a training environment associated with the real-world sports environment (See Dooley: Fig. 1, and [0064], "The preferred embodiment has both solo and multi-player operational modes: User Directed Mode--this mode allows the athlete to determine the types of movements to undertake while receiving selected feedback that may include coaching tips and performance feedback. With this mode, the athlete may elect to introduce interactivity and spontaneity by having a real world training partner interact with her in the same physical space. Or the athlete can elect to receive feedback or coaching tips while training with a conventional cone drill. The selected feedback may include reaction time, 1st step quickness, depth of stance, velocity, caloric expenditure, etc. that would not be measurable with a stopwatch measuring only elapsed time. Device Program Mode--this mode has the device delivering pre-programmed training protocols to which the athlete responds and receives selected feedback. Single User Mode--both the aforementioned modes are single player modes. Multiplayer Mode--this mode provides for two-way, realtime interaction between two or more athletes within the same physical space. The objective of multiplayer activities is to introduce interactivity and spontaneity for more realistic training. The ever-changing spatial relationship between the athletes creates a competitive or cooperative experience that can realistically approximate actual game play in reaction-based sports").
Regarding claim 44, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Dooley teaches that the system of claim 28, wherein the operations further comprise enabling coaches to create simulated game plays using the physical markers (See Dooley: Fig. 2, and [0066], "The prime objective of the present invention is to "monitor" the athlete during locomotion in order to detect kinematic or physical performance factors that may expose the athlete to an increased risk of injury or that negatively impact the athlete's performance capabilities. At such times as the athlete's kinematics and/or physical performance are maintained within predefined acceptable limits, the athlete can be rewarded with positive feedback. However, at such times when the athlete's movement exceeds the pre-established acceptable limits, cautionary feedback may be delivered to the athlete. Certain performance ranges can be established or adjusted based on the athlete's anthropometrics, age, medical history, sport of interest, fitness level, etc. By way of example, the present invention may be programmed for "acceptable ranges" relating to the preferred depth of the athlete's stance. For example, a desired depth of stance for a certain athlete may be in a range from minus 8 inches to minus 14 inches as measured from her standing height. Feedback in the form of coaching tips (advice) can also be delivered. The feedback may be aural, tactile and/or by visual or other suitable means").
Regarding claim 45, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Dooley teaches that the system of claim 28, wherein the operations further comprise generating AR overlays indicating interaction among the physical markers (See Dooley: Fig. 7, and [0080], "Video see-through HMDs use cameras mounted near the user's head/eye region to take video images of the real world and feed them back to a computing system. The computing system can then take the captured images of the real world and overlay or embed the virtual objects into each frame of video to form a composite image. This new sequence of images or video is then projected back to the HMD for viewing by the user. A known deficit with video see-through HMDs is the time lag associated with capturing, processing and displaying the augmented images; all of which can cause the user to experience a delay in viewing the images. As technology improves, this delay will be become less noticeable. Until the optical see-through HMDs are readily available, the video see-through HMDs are implemented for the preferred embodiment of the current invention. An example of a video see-through eyewear is the Vuzix WRAP 920AR, an HMD that incorporates motion tracking").
Regarding claim 46, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Muthyala teaches that the system of claim 28, wherein the operations further comprise selecting an option associated with the at least one interconnectable AR object primitive based on a user interaction (See Muthyala: Fig. 2, and [0027], "In some embodiments, a marker construction element may comprise two different AR markers and a movable element, e.g. a hinged element, a rotatably arranged element, or the like, that may be moved between a first and a second position, such that the movable element selectively obstructs a respective one of the markers from view when positioned in the first and second position, respectively. Hence, movement of the component causes different markers to become visible, thus allowing the AR computer system to detect the state of the movable component").
Regarding claim 47, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Muthyala teaches that the system of claim 28, wherein the operations further comprise adjusting the AR sports content set automatically based on the arrangement of the physical markers (See Muthyala: Figs. 1A-D, and Figs. 9A-B, and [0049], "FIGS. 1 a-d each show a prior art toy building brick with coupling studs 105 on their top surface and a cavity 102 extending into the brick from the bottom. FIGS. la-b show a top side of a toy building brick, while FIG. 1b shows the bottom side of the same toy building brick. FIGS. lc-d show examples of similar toy building bricks of different sizes. Such construction elements are widely available under the trade name LEGO. The cavity has a central tube 103, and coupling studs of another brick can be received in the cavity in a frictional engagement as disclosed in U.S. Pat. No. 3,005,282. The building bricks shown in the remaining figures may have this known type of coupling means in the form of cooperating studs and cavities. However, other types of coupling means may also be used. The coupling studs are arranged in a square planar grid, i.e. defining orthogonal directions along which sequences of coupling studs are arranged. Generally, such an arrangement of coupling elements allows the toy bricks to be interconnected in a discrete number of orientations relative two each other, in particular at right angles with respect to each other. It will be appreciated that other geometric arrangements of coupling elements may result in different orientational constraints. For example, the coupling elements may be arranged in a triangular, regular grid allowing a building element to be placed on another building element in three different orientations"; and [0091], "It will be appreciated that the marker construction elements may also trigger further actions within the AR system. For example, the landscape of FIG. 9b may be used in the AR system as a landscape within which a game is performed, i.e. the user may build his/her own levels of a computer game by using physical building elements". Note that the AR marker arrangements in the real world environment triggers the AR scene generation automatically adapted to the AR marker arrangements).
Regarding claim 48, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Muthyala, Dooley, Carignano and Gagner teach that a computer-implemented method of supporting an augmented reality (AR) sport event, the method (See Muthyala: Fig. 2, and [0050], "FIG. 2 shows an embodiment of a toy construction system as disclosed herein. The system comprises a computer 15, a display 1, a video camera 5, a toy construction model 10 constructed from at least one toy construction element and at least one marker construction element 11 of a toy construction system as described herein. The computer may be a personal computer, a desktop computer, a laptop computer, a hand held computer, a game console, a handheld entertainment device, or any other suitably programmable computer. The display 1 is operatively coupled to (e.g. integrated into) the computer 15, and operable to display, under the control of the computer 15, a video image. It will be appreciated that the system may comprise further peripheral computer devices operatively coupled to the computer 15, such as a keyboard, a mouse 2 or other pointing device, and/or the like. The video camera 5 is operable to capture video images of the environment in which the video camera is situated, e.g. of a field of view 7 including a visible surface area 8 of a surface 4, e.g. a desktop, a floor, or the like.
Hence the visible surface area 8 is the projection of the field of view of the video camera on the surface 4. The video camera is operatively coupled to (e.g. integrated into) the computer 15 and operable to forward the captured video image to a processing unit of the computer 15, e.g. via a suitable input interface of the computer 15. For example, the video camera may be a webcam connected to or integrated into computer 15. In the example of FIG. 2, the video camera is positioned on a camera support 6, such as a tripod, supporting the camera at a predetermined height above the surface 4") comprising:
receiving a digital representation of an arrangement of physical markers in a real-world sports environment (See Muthyala: Figs. 1-3, and [0056], "The computer 15 is operable to detect the presence of the two-dimensional machine readable code in the captured image. Furthermore, the computer 15 may determine a relative position and orientation of the two-dimensional machine readable code relative to the position of the video camera 15"; [0057], "Accordingly, the computer 15 may modify the captured image of the toy construction model resulting in a modified video image 12 displayed on the display 1. In this example, the modified image shows the captured image 13 of the toy construction model and computer-generated image elements 14 superimposed the captured image. The computer generates the computer-generated image elements at a position and orientation within the video image 12 as determined from the detected position and orientation of the AR marker of the marker construction element 11"; and [0063], "The AR marker may be arranged in a uniform manner relative to the coupling means, i.e. to the coupling studs on the top surface and/or to the coupling cavity in the bottom. For example, the AR marker may define a direction parallel or normal to the planar grid defined by the coupling means. This makes the marker bricks interchangeable, and in a toy structure built from bricks as in FIGS. 1-3, several marker bricks can be used interchangeably, and a particular marker brick can be used in several constructions. A toy construction system may comprise several of such marker bricks having different insignia applied to it and causing the computer system to generate different computer-generated images. Nevertheless, if all marker bricks include the insignia at uniform positions, such marker bricks may easily be interchanged within a toy construction built from the building bricks described herein". Note that the 2D machine readable code is mapped to the digital representation);
identifying each marker's identity from digital features derived from the digital representation (See Muthyala: Figs. SA-G, and [0067], "FIGS. Sa-g illustrate examples of composite marker construction elements, i.e. a plurality of marker construction elements directly or indirectly connected with each other by means of the coupling means of the toy construction system, e.g. in a predetermined spatial relationship with each other. Generally, composite marker construction elements allow the user to create a large set of distinct objects, each identifiable by one of a large number of unique AR markers. In particular, the large number of unique AR markers may be created from a relatively limited set of individual marker construction elements, because combining a smaller set of markers through a building system dramatically increases the total number of combinatorial possibilities");
identifying at least one interconnectable AR object primitive for each identified marker (See Muthyala: Fig. 1-5, and [0065], "FIG. 4 illustrates how a marker construction element may be coupled to another toy construction element of a toy construction model. In this example, the toy construction model comprises toy construction elements 410 and 412 and a marker construction element 411. FIG. 4a shows the individual constructions elements, while FIG. 4b shows the marker construction element 411 releasably coupled to the construction element 410. To this end, the toy construction element 410 comprises coupling studs 405 on its top surface as described in connection with FIG. 1. The marker construction element 404 comprises cavities on its bottom surface (not shown in FIG. 4), allowing it to frictionally engage the coupling studs 404"); and [0066], "Hence, the user may connect a marker construction element comprising an AR marker to a constructed toy model so as to facilitate tracking the toy model in 3D space and to superimpose virtual graphics and effects, thereby providing the physical toy with virtual attributes and allow the constructed toy model to become a part of an AR application. As in the example of FIG. 3, the insignia of the marker construction element 411 is located on a surface of the marker construction element that does not comprise coupling element". Note that the physical pieces with markers is a specific virtual model, and each marker (on the physical blocks, construction elements) maps to a virtual 3D object or "primitive", and the physical/virtual coupling rules are mapped to "interconnectable"), the at least one interconnectable AR object primitive comprising at least one primitive interface through which the at least one interconnectable AR object primitive interacts with other interconnectable AR object primitives (See Carignano: Figs. 1A-D, and [0046], “In operation, the camera 110 creates image data of the environment 120, wherein preferably the image data may correspond to a dynamic state of the environment 120 which may, for instance, be represented by merely moving the camera 110 with respect to the environment 120, or by providing moveable objects within the environment, for instance the marker objects 125, 126, or one or more of the objects 121 . . . 123 may be moveable. For example, the point of view of the environment 120 may be changed by moving around the camera 110 within the environment 120, thereby allowing to observe especially the marker objects 125, 126 from different perspectives so as to enable the assessment of virtual objects created by the generator 150 from different points of view. The image data provided by the camera 110 which may continuously be updated, are received by the identifying means 130, which recognizes the marker objects 125, 126 and enables the tracking of the marker objects 125, 126 once they are identified, even if pattern recognition is hampered by continuously changing the point of view by, for instance, moving the camera 110 or the marker objects 125, 126. For example, after identifying a predefined pattern associated with the marker objects 125, 126 within the image data, the identifying means 130 may inform the combining means 140 about the presence of a marker object within a specified image data area and based on this information, the means 140 may then continuously track the corresponding object represented by the image data used for identifying the marker objects 125, 126, assuming that the marker objects 125, 126 will not vanish over time. In other embodiments, the process of identifying the marker objects 125, 126 may be performed substantially continuously or at least may be repeated on a regular basis so as to confirm the presence of the marker objects 125, 126 and also to verify or enhance the tracking accuracy of the combining means 140. Based on the image data of the environment and the information provided by the identifying means 130, the combining means 140 creates three-dimensional image data and superimposes corresponding three-dimensional image data received from the object generator 150, wherein the three-dimensional object data are permanently updated on the basis of the tracking operation of the means 140. For instance, the means 140 may, based on the information of the identifying means 130, calculate the position of the camera 110 with respect to the marker objects 125, 126 and use this coordinate information for determining the coordinates of a virtual camera, thereby allowing a precise "overlay" of the object data delivered by the generator 150 with the image data of the marker objects 125, 126. The coordinate information also includes data on the relative orientation of the marker objects 125, 126 with respect to the camera 110, thereby enabling the combining means 140 to correctly adapt the orientation of the virtual object. Finally, the combined three-dimensional virtual image data may be presented by the output means 160 in any appropriate form. For example, the output means 160 may comprise appropriate display means so as to visualize the environment 120 including virtual objects associated with the marker objects 125, 126. When operating the system 100 it is advantageous to pre-install recognition criteria for at least one marker object 125, 126 so as to allow a substantially reliable real-time image processing. Moreover, the correlation between a respective marker object and one or more virtual objects may be established prior to the operation of the system 100 or may also be designed so as to allow an interactive definition of an assignment of virtual objects to marker objects. For example, upon user request, virtual objects initially assigned to the marker object 125 may be assigned to the marker object 126 and vice versa. Moreover, a plurality of virtual objects may be assigned to a single marker object and a respective one of the plurality of virtual objects may be selected by the user, by a software application, and the like”; and [0056], “That is, after gathering image data by the optical imaging system 310, a marker object is identified, for instance attached to a user positioned so as to face the optical imaging system 310 and the display surface 360, and the marker object is identified and tracked, as is previously described. Moreover, based on a predefined or user-initiated association of one or more marker objects with respective virtual objects, the generator 350 provides corresponding object data to the combining means 340, which may produce corresponding virtual image data including the virtual object”. Note that the marker object and the virtual object is predefined or user-initiated, and this is mapped to the interconnectable AR object primitive);
generating an AR sports content set (See Dooley: Fig. 1-7, and [0094], "Referring to FIG. 7, the wearable display 220 depicts real world images seen through the glasses 220 that include three trees, and virtual reality cues overlaid on the real world images. The virtual reality depicts a start and a racing hurdle on the right glass and an arrow on the left glass. The arrow tells the user that she must jump higher to clear the hurdle. Although the right and left glasses show different images, the user sees the three trees, hurdle and arrow as a single display"; and [0096], "In step 320, the personal computer 111 reads the three dimensional information from the source 110 and uses transmitter 112 to transmit the information wirelessly to receiver 103. This step is necessary because the active controller 101 transmits the data directly to the source unit 110. If the transmission protocol were known and was able to be mimicked by the body worn computing device 102, this step would not be needed, as the computing device 102 could simply communicate with the active controller 101 directly. In step 325, the computing device 102 generates the virtual simulation using the positional and orientation data from the passive controllers l00A-F and displays the information on the wearable display 120". Note that the generated AR sports content is common application of AR system in virtual game field) by coupling at least two AR object primitives according to a sports application-specific rules set and the arrangement of the physical markers (See Muthyala: Figs. 9A-B, and [0090], "FIG. 9 shows toy construction elements 911a-911b of the type shown in FIG. 1. In this embodiment, the different construction elements have respective insignia, different color, and/or another distinguishable visual feature. Each of the toy construction element, when detected by the AR system within a toy construction model, triggers a predetermined action. For example, in the example of FIG. 9, each of the construction elements 911a-911d represents a different type of landscape or environmental phenomena, such as gras, water, snow, fire, etc. When detected by the AR system, each construction element may thus be augmented by the AR system by displaying corresponding image elements representing grass, water, fire, snow, respectively. This allows a user to build a landscape by means of the construction elements, e.g. as illustrated in gig. 9b"), wherein the AR sports content set is generated at least in part based on an interaction between the at least two AR object primitives through the at least one primitive interface (See Gagner: Figs. 1-2 and [0021], “In one implementation, at stage A, the gaming machine 160 detects a fiducial marker in one or more images captured by a camera 165 of the gaming machine 160. In one example, a player positions a game ticket 122 (or other object) including a fiducial marker 124 in the field of vision of the camera 165. The camera 165 captures video of objects in the field of vision of the camera 165 including the game ticket 122 with the fiducial marker 124. In this example, an image processing mechanism of the gaming machine 160 detects the fiducial marker 124 on the game ticket 122 captured in the video. In one implementation, the fiducial marker 124 can include a fiducial code 125 and a bounding indicator 126. In some examples, the fiducial code 125 may include a 1D barcode, 2D barcode, geometric patterns, text, or a combination thereof, that can be used to identify which augmented reality object to render. The fiducial code 125 may also include additional metadata and other information associated with the fiducial marker, e.g., a serial number to track when the player uses the fiducial marker. In one example, the bounding indicator 126 may include a bounding square or similar indicator that surrounds the fiducial code 125 and helps to identify the location of the fiducial code 125. The bounding indicator 126 can also indicate the orientation of the fiducial marker 124, as will be further described below”; [0027], “At stage F, the wagering game server 150 determines how to render the augmented reality object based on the fiducial code 125 and the orientation of the fiducial marker 124. For example, metadata within the fiducial code 125 and/or the orientation of the fiducial marker 124 can determine attributes of an augmented reality 3D object, e.g., determine movement, orientation, composition, etc. of the augmented reality 3D object”; [0028], “At stage G, the wagering game server 150 composites the augmented reality object with the wagering game content that is being provided to the gaming machine 160. In some instances, compositing can include combining visual elements from separate sources into single images, e.g., one or more images that comprise a video stream. In one implementation, the wagering game server 150 composites video of the fiducial marker 124 and the augmented reality object with the wagering game content. In some implementations, the wagering game server 150 uses the fiducial marker 124 as a reference point when performing the compositing operations. It is noted, however, that in other implementations the gaming machine 160 can be configured to perform the compositing operations. In these implementations, the wagering game server 150 can provide the wagering game content and the rendered augmented reality object to the gaming machine 160, and the gaming machine 160 can composite the video of the fiducial marker 124 and the augmented reality object with the wagering game content”; and [0031], “FIG. 2 is a conceptual diagram illustrating an example of using the fiducial marker 124 to control various attributes of an augmented reality 3D object, according to some embodiments. In the example shown in FIG. 2, during a wagering game session, the gaming machine 160 presents on the display device 166 wagering game content composited with video of the fiducial marker 124 and the associated augmented reality 3D object. In one implementation, the player can rotate, tilt, or otherwise change the orientation of the fiducial mark 124 (while the fiducial marker 124 is positioned in the field of vision of the camera 165) to control various attributes of the augmented reality 3D object during game play. For example, modifying the orientation of the fiducial marker 124 can control the movement, modify the orientation, change the composition, etc. of the augmented reality 3D object, and allow the player to interact with and play the wagering game. In the example of FIG. 2, the wagering game may be a secondary bonus game (e.g., a picking game) and the augmented reality 3D object is an avatar 175 holding a flashlight. In this example, modifying the orientation of the fiducial marker 124 can control the direction the avatar 175 walks within the game, control the direction the avatar 175 points the flashlight, change the orientation of the avatar 175 and flashlight with respect to the point of view of the player, control what other tools (besides the flashlight) the avatar 175 uses, controls other movements of the avatar 175 (e.g., opening doors), etc. It is noted that in other examples various types of augmented reality objects can be used to play various types of wagering games in a similar manner”. Note that detecting the fiducial markers, rendering the specific AR objects based on the marker identity, compositing the AR objects into the scene, and controlling attributes of the AR object is mapped to the AR sports content set generated based on the interaction between the AR objects primitives associated with markers through the primitive interface. The teachings of Carignano and Gagner provide interactive, rule-driven behaviors of marker-associated interconnectable AR objects and generating AR content set based on the interactions between the AR object); and
causing a device to present the AR sports content set superimposed on an image of the real-world sports environment (See Muthyala: Fig. 9A-B, and [0006], "Within the different technical field of image recognition and computer graphics, systems that utilize augmented reality (AR) have attracted increased interest. Augmented reality (AR) is a technology where a captured live view of one or more items of a physical, real-world environment is augmented by a computer-generated graphics, i.e. where a user is presented with a composite view comprising the live view and the computer-generated graphics, e.g. in the form of an overlay/superposition of the live view with the computer-graphics"; and [0091], "It will be appreciated that the marker construction elements may also trigger further actions within the AR system. For example, the landscape of FIG. 9b may be used in the AR system as a landscape within which a game is performed, i.e. the user may build his/her own levels of a computer game by using physical building elements").
Regarding claim 49, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Muthyala, Dooley, Carignano and Gagner teach that a computer program product including one or more non-transitory computer readable media storing software instructions that, when executed, cause a processor to perform operations for supporting an augmented reality (AR) sport event, the operations (See Muthyala: Fig. 2, and [0050], "FIG. 2 shows an embodiment of a toy construction system as disclosed herein. The system comprises a computer 15, a display 1, a video camera 5, a toy construction model 10 constructed from at least one toy construction element and at least one marker construction element 11 of a toy construction system as described herein. The computer may be a personal computer, a desktop computer, a laptop computer, a handheld computer, a game console, a handheld entertainment device, or any other suitably programmable computer. The display 1 is operatively coupled to (e.g. integrated into) the computer 15, and operable to display, under the control of the computer 15, a video image. It will be appreciated that the system may comprise further peripheral computer devices operatively coupled to the computer 15, such as a keyboard, a mouse 2 or other pointing device, and/or the like. The video camera 5 is operable to capture video images of the environment in which the video camera is situated, e.g. of a field of view 7 including a visible surface area 8 of a surface 4, e.g. a desktop, a floor, or the like. Hence the visible surface area 8 is the projection of the field of view of the video camera on the surface 4. The video camera is operatively coupled to (e.g. integrated into) the computer 15 and operable to forward the captured video image to a processing unit of the computer 15, e.g. via a suitable input interface of the computer 15. For example, the video camera may be a webcam connected to or integrated into computer 15. In the example of FIG. 2, the video camera is positioned on a camera support 6, such as a tripod, supporting the camera at a predetermined height above the surface 4") comprising:
receiving a digital representation of an arrangement of physical markers in a real-world sports environment (See Muthyala: Figs. 1-3, and [0056], "The computer 15 is operable to detect the presence of the two-dimensional machine readable code in the captured image. Furthermore, the computer 15 may determine a relative position and orientation of the two-dimensional machine readable code relative to the position of the video camera 15"; [0057], "Accordingly, the computer 15 may modify the captured image of the toy construction model resulting in a modified video image 12 displayed on the display 1. In this example, the modified image shows the captured image 13 of the toy construction model and computer-generated image elements 14 superimposed the captured image. The computer generates the computer-generated image elements at a position and orientation within the video image 12 as determined from the detected position and orientation of the AR marker of the marker construction element 11"; and [0063], "The AR marker may be arranged in a uniform manner relative to the coupling means, i.e. to the coupling studs on the top surface and/or to the coupling cavity in the bottom. For example, the AR marker may define a direction parallel or normal to the planar grid defined by the coupling means. This makes the marker bricks interchangeable, and in a toy structure built from bricks as in FIGS. 1-3, several marker bricks can be used interchangeably, and a particular marker brick can be used in several constructions. A toy construction system may comprise several of such marker bricks having different insignia applied to it and causing the computer system to generate different computer-generated images. Nevertheless, if all marker bricks include the insignia at uniform positions, such marker bricks may easily be interchanged within a toy construction built from the building bricks described herein". Note that the 2D machine readable code is mapped to the digital representation);
identifying each marker's identity from digital features derived from the digital representation (See Muthyala: Figs. SA-G, and [0067], "FIGS. Sa-g illustrate examples of composite marker construction elements, i.e. a plurality of marker construction elements directly or indirectly connected with each other by means of the coupling means of the toy construction system, e.g. in a predetermined spatial relationship with each other. Generally, composite marker construction elements allow the user to create a large set of distinct objects, each identifiable by one of a large number of unique AR markers. In particular, the large number of unique AR markers may be created from a relatively limited set of individual marker construction elements, because combining a smaller set of markers through a building system dramatically increases the total number of combinatorial possibilities");
identifying at least one interconnectable AR object primitive for each identified marker (See Muthyala: Fig. 1-5, and [0065], "FIG. 4 illustrates how a marker construction element may be coupled to another toy construction element of a toy construction model. In this example, the toy construction model comprises toy construction elements 410 and 412 and a marker construction element 411. FIG. 4a shows the individual constructions elements, while FIG. 4b shows the marker construction element 411 releasably coupled to the construction element 410. To this end, the toy construction element 410 comprises coupling studs 405 on its top surface as described in connection with FIG. 1. The marker construction element 404 comprises cavities on its bottom surface (not shown in FIG. 4), allowing it to frictionally engage the coupling studs 404"); and [0066], "Hence, the user may connect a marker construction element comprising an AR marker to a constructed toy model so as to facilitate tracking the toy model in 3D space and to superimpose virtual graphics and effects, thereby providing the physical toy with virtual attributes and allow the constructed toy model to become a part of an AR application. As in the example of FIG. 3, the insignia of the marker construction element 411 is located on a surface of the marker construction element that does not comprise coupling element". Note that the physical pieces with markers is a specific virtual model, and each marker (on the physical blocks, construction elements) maps to a virtual 3D object or "primitive", and the physical/virtual coupling rules are mapped to "interconnectable"), the at least one interconnectable AR object primitive comprising at least one primitive interface through which the at least one interconnectable AR object primitive interacts with other interconnectable AR object primitives (See Carignano: Figs. 1A-D, and [0046], “In operation, the camera 110 creates image data of the environment 120, wherein preferably the image data may correspond to a dynamic state of the environment 120 which may, for instance, be represented by merely moving the camera 110 with respect to the environment 120, or by providing moveable objects within the environment, for instance the marker objects 125, 126, or one or more of the objects 121 . . . 123 may be moveable. For example, the point of view of the environment 120 may be changed by moving around the camera 110 within the environment 120, thereby allowing to observe especially the marker objects 125, 126 from different perspectives so as to enable the assessment of virtual objects created by the generator 150 from different points of view. The image data provided by the camera 110 which may continuously be updated, are received by the identifying means 130, which recognizes the marker objects 125, 126 and enables the tracking of the marker objects 125, 126 once they are identified, even if pattern recognition is hampered by continuously changing the point of view by, for instance, moving the camera 110 or the marker objects 125, 126. For example, after identifying a predefined pattern associated with the marker objects 125, 126 within the image data, the identifying means 130 may inform the combining means 140 about the presence of a marker object within a specified image data area and based on this information, the means 140 may then continuously track the corresponding object represented by the image data used for identifying the marker objects 125, 126, assuming that the marker objects 125, 126 will not vanish over time. In other embodiments, the process of identifying the marker objects 125, 126 may be performed substantially continuously or at least may be repeated on a regular basis so as to confirm the presence of the marker objects 125, 126 and also to verify or enhance the tracking accuracy of the combining means 140. Based on the image data of the environment and the information provided by the identifying means 130, the combining means 140 creates three-dimensional image data and superimposes corresponding three-dimensional image data received from the object generator 150, wherein the three-dimensional object data are permanently updated on the basis of the tracking operation of the means 140. For instance, the means 140 may, based on the information of the identifying means 130, calculate the position of the camera 110 with respect to the marker objects 125, 126 and use this coordinate information for determining the coordinates of a virtual camera, thereby allowing a precise "overlay" of the object data delivered by the generator 150 with the image data of the marker objects 125, 126. The coordinate information also includes data on the relative orientation of the marker objects 125, 126 with respect to the camera 110, thereby enabling the combining means 140 to correctly adapt the orientation of the virtual object. Finally, the combined three-dimensional virtual image data may be presented by the output means 160 in any appropriate form. For example, the output means 160 may comprise appropriate display means so as to visualize the environment 120 including virtual objects associated with the marker objects 125, 126. When operating the system 100 it is advantageous to pre-install recognition criteria for at least one marker object 125, 126 so as to allow a substantially reliable real-time image processing. Moreover, the correlation between a respective marker object and one or more virtual objects may be established prior to the operation of the system 100 or may also be designed so as to allow an interactive definition of an assignment of virtual objects to marker objects. For example, upon user request, virtual objects initially assigned to the marker object 125 may be assigned to the marker object 126 and vice versa. Moreover, a plurality of virtual objects may be assigned to a single marker object and a respective one of the plurality of virtual objects may be selected by the user, by a software application, and the like”; and [0056], “That is, after gathering image data by the optical imaging system 310, a marker object is identified, for instance attached to a user positioned so as to face the optical imaging system 310 and the display surface 360, and the marker object is identified and tracked, as is previously described. Moreover, based on a predefined or user-initiated association of one or more marker objects with respective virtual objects, the generator 350 provides corresponding object data to the combining means 340, which may produce corresponding virtual image data including the virtual object”. Note that the marker object and the virtual object is predefined or user-initiated, and this is mapped to the interconnectable AR object primitive);
generating an AR sports content set (See Dooley: Fig. 1-7, and [0094], "Referring to FIG. 7, the wearable display 220 depicts real world images seen through the glasses 220 that include three trees, and virtual reality cues overlaid on the real world images. The virtual reality depicts a start and a racing hurdle on the right glass and an arrow on the left glass. The arrow tells the user that she must jump higher to clear the hurdle. Although the right and left glasses show different images, the user sees the three trees, hurdle and arrow as a single display"; and [0096], "In step 320, the personal computer 111 reads the three dimensional information from the source 110 and uses transmitter 112 to transmit the information wirelessly to receiver 103. This step is necessary because the active controller 101 transmits the data directly to the source unit 110. If the transmission protocol were known and was able to be mimicked by the body worn computing device 102, this step would not be needed, as the computing device 102 could simply communicate with the active controller 101 directly. In step 325, the computing device 102 generates the virtual simulation using the positional and orientation data from the passive controllers l00A-F and displays the information on the wearable display 120". Note that the generated AR sports content is common application of AR system in virtual game field) by coupling at least two AR object primitives according to a sports application-specific rules set and the arrangement of the physical markers (See Muthyala: Figs. 9A-B, and [0090], "FIG. 9 shows toy construction elements 911a-911b of the type shown in FIG. 1. In this embodiment, the different construction elements have respective insignia, different color, and/or another distinguishable visual feature. Each of the toy construction element, when detected by the AR system within a toy construction model, triggers a predetermined action. For example, in the example of FIG. 9, each of the construction elements 911a-911d represents a different type of landscape or environmental phenomena, such as gras, water, snow, fire, etc. When detected by the AR system, each construction element may thus be augmented by the AR system by displaying corresponding image elements representing grass, water, fire, snow, respectively. This allows a user to build a landscape by means of the construction elements, e.g. as illustrated in gig. 9b"), wherein the AR sports content set is generated at least in part based on an interaction between the at least two AR object primitives through the at least one primitive interface (See Gagner: Figs. 1-2 and [0021], “In one implementation, at stage A, the gaming machine 160 detects a fiducial marker in one or more images captured by a camera 165 of the gaming machine 160. In one example, a player positions a game ticket 122 (or other object) including a fiducial marker 124 in the field of vision of the camera 165. The camera 165 captures video of objects in the field of vision of the camera 165 including the game ticket 122 with the fiducial marker 124. In this example, an image processing mechanism of the gaming machine 160 detects the fiducial marker 124 on the game ticket 122 captured in the video. In one implementation, the fiducial marker 124 can include a fiducial code 125 and a bounding indicator 126. In some examples, the fiducial code 125 may include a 1D barcode, 2D barcode, geometric patterns, text, or a combination thereof, that can be used to identify which augmented reality object to render. The fiducial code 125 may also include additional metadata and other information associated with the fiducial marker, e.g., a serial number to track when the player uses the fiducial marker. In one example, the bounding indicator 126 may include a bounding square or similar indicator that surrounds the fiducial code 125 and helps to identify the location of the fiducial code 125. The bounding indicator 126 can also indicate the orientation of the fiducial marker 124, as will be further described below”; [0027], “At stage F, the wagering game server 150 determines how to render the augmented reality object based on the fiducial code 125 and the orientation of the fiducial marker 124. For example, metadata within the fiducial code 125 and/or the orientation of the fiducial marker 124 can determine attributes of an augmented reality 3D object, e.g., determine movement, orientation, composition, etc. of the augmented reality 3D object”; [0028], “At stage G, the wagering game server 150 composites the augmented reality object with the wagering game content that is being provided to the gaming machine 160. In some instances, compositing can include combining visual elements from separate sources into single images, e.g., one or more images that comprise a video stream. In one implementation, the wagering game server 150 composites video of the fiducial marker 124 and the augmented reality object with the wagering game content. In some implementations, the wagering game server 150 uses the fiducial marker 124 as a reference point when performing the compositing operations. It is noted, however, that in other implementations the gaming machine 160 can be configured to perform the compositing operations. In these implementations, the wagering game server 150 can provide the wagering game content and the rendered augmented reality object to the gaming machine 160, and the gaming machine 160 can composite the video of the fiducial marker 124 and the augmented reality object with the wagering game content”; and [0031], “FIG. 2 is a conceptual diagram illustrating an example of using the fiducial marker 124 to control various attributes of an augmented reality 3D object, according to some embodiments. In the example shown in FIG. 2, during a wagering game session, the gaming machine 160 presents on the display device 166 wagering game content composited with video of the fiducial marker 124 and the associated augmented reality 3D object. In one implementation, the player can rotate, tilt, or otherwise change the orientation of the fiducial mark 124 (while the fiducial marker 124 is positioned in the field of vision of the camera 165) to control various attributes of the augmented reality 3D object during game play. For example, modifying the orientation of the fiducial marker 124 can control the movement, modify the orientation, change the composition, etc. of the augmented reality 3D object, and allow the player to interact with and play the wagering game. In the example of FIG. 2, the wagering game may be a secondary bonus game (e.g., a picking game) and the augmented reality 3D object is an avatar 175 holding a flashlight. In this example, modifying the orientation of the fiducial marker 124 can control the direction the avatar 175 walks within the game, control the direction the avatar 175 points the flashlight, change the orientation of the avatar 175 and flashlight with respect to the point of view of the player, control what other tools (besides the flashlight) the avatar 175 uses, controls other movements of the avatar 175 (e.g., opening doors), etc. It is noted that in other examples various types of augmented reality objects can be used to play various types of wagering games in a similar manner”. Note that detecting the fiducial markers, rendering the specific AR objects based on the marker identity, compositing the AR objects into the scene, and controlling attributes of the AR object is mapped to the AR sports content set generated based on the interaction between the AR objects primitives associated with markers through the primitive interface. The teachings of Carignano and Gagner provide interactive, rule-driven behaviors of marker-associated interconnectable AR objects and generating AR content set based on the interactions between the AR object); and
causing a device to present the AR sports content set superimposed on an image of the real-world sports environment (See Muthyala: Fig. 9A-B, and [0006], "Within the different technical field of image recognition and computer graphics, systems that utilize augmented reality (AR) have attracted increased interest. Augmented reality (AR) is a technology where a captured live view of one or more items of a physical, real-world environment is augmented by a computer-generated graphics, i.e. where a user is presented with a composite view comprising the live view and the computer-generated graphics, e.g. in the form of an overlay/superposition of the live view with the computer-graphics"; and [0091], "It will be appreciated that the marker construction elements may also trigger further actions within the AR system. For example, the landscape of FIG. 9b may be used in the AR system as a landscape within which a game is performed, i.e. the user may build his/her own levels of a computer game by using physical building elements").
Claim 33 is rejected under 35 U.S.C. 103 as being unpatentable over Muthyala, etc. (US 20140378023 A1), in view of Dooley, etc. (US 20110270135 A1), further in view of Carignano, etc. (US 20070038944 A1), Gagner, etc. (US 20110065496 A1) and Miyashita, etc. (US 20130207894 A1).
Regarding claim 33, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. However, Muthyala, modified by Dooley, Carignano and Gagner, fails to explicitly disclose that the system of claim 28, wherein the operations further comprise recognizing non-marker objects in the real-world sports environment and incorporating representations of the non-marker objects into the AR sports content set.
However, Miyashita teaches that the system of claim 28, wherein the operations further comprise recognizing non-marker objects in the real-world sports environment and incorporating representations of the non-marker objects into the AR sports content set (See Miyashita: Fig. 4, and [0047], "FIG. 4 is a flowchart illustrating algorithm for determining information content (zoom ratio) displayed on the mobile device 200 according to relative positional relationship between the display device 100 and mobile device 200. At first, the mobile device 200 captures an image of an appearance in front thereof (display part 110 of the display device 100) using the camera 240 (step Sl0). The display device 100 displays an explicit marker such as a QR code or a beforehand registered non-marker object on the screen. Herein, the beforehand registered non-marker object is a marker capable of being recognized by a marker-less recognizer, for example. As one example, the non-marker object can be a figure of 2 cm square. The mobile device 200 detects the above-mentioned marker or non-marker object from the captured image (step S12). Then, the mobile device 200 detects relative positional relationship between the display device 100 and mobile device 200 based on the detected marker or non-marker object (step S14), and furthermore, detects the distance (step S16). The distance/position information acquisition part 220a of the CPU 220 beforehand acquires the size and shape of the marker or non-marker object, and thereby, can acquire the relative positional relationship and distance between the display device 100 and mobile device 200 based on information of the marker or non-marker object whose image is captured by the image capturing part 240. Next, the mobile device 200 dynamically adjusts a graph illustrated in FIG. 5 (step S18). The mobile device 200 calculates a zoom ratio in displaying information according to the distance between the detected two devices (step S20). Finally, the mobile device 200 displays the information according to the calculated zoom ratio (step S22)").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Muthyala to have the system of claim 28, wherein the operations further comprise recognizing non-marker objects in the real-world sports environment and incorporating representations of the non-marker objects into the AR sports content set as taught by Miyashita in order to provide the information efficiently for the user (See Miyashita: Fig. 3, and [0063], "As described above, according to the embodiment, display status of the display part 210 is changed in a non-linear manner in response to movement of the mobile device 200, and therefore, portions of importance for the user can be displayed intensively. Accordingly, information can be efficiently provided for the user"). Muthyala teaches a method and system that may generate AR content using the physical markers arranged in the real word environment to identify and configure marker based interconnectable virtual primitives (building blocks, scene, trees) with rules sets for virtual primitive arrangements; while Miyashita teaches a system and method that may generate AR content with marker-based object recognition and non-marker register based objects. Therefore, it is obvious to one of ordinary skill in the art to modify Muthyala by Miyashita to mix the AR content of the marker based and the non-marker (marker-less) objects. The motivation to modify Muthyala by Miyashita is "Use of known technique to improve similar devices (methods, or products) in the same way".
Claims 39 and 50 are rejected under 35 U.S.C. 103 as being unpatentable over Muthyala, etc. (US 20140378023 A1), in view of Dooley, etc. (US 20110270135 A1), further in view of Carignano, etc. (US 20070038944 A1), Gagner, etc. (US 20110065496 A1) and Belimpasakis, etc. (US 20110238751 A1).
Regarding claim 39, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. However, Muthyala, modified by Dooley, Carignano and Gagner, fails to explicitly disclose that the system of claim 28, wherein the AR sports content set includes virtual advertising content positioned within the real-world sports environment.
However, Belimpasakis teaches that the system of claim 28, wherein the AR sports content set includes virtual advertising content positioned within the real-world sports environment (See Belimpasakis: Figs. 2A-F, and [0045], "The control logic 221 may also periodically broadcast decoy queries and replies to make tracking an individual wireless node 101 more difficult. Since an outside observer does not know the authentication key associated with a community, the observer cannot distinguish a valid message from a fictitious one. Accordingly, by observing decoy messages, the observer is likely to detect presence of a private community when there is not one. Additionally, the control logic 221 enables to user to define filters for incoming information (e.g., filter advertisements) and how these filters would work (e.g., ignore the information completely, relay the information but do not store, etc.). It is also contemplated that the user can direct the control logic 221 to control the user's visibility on the ad-hoc mesh network 109 (e.g., no visibility, visible only to a certain community or other user) to maintain privacy. As another mechanism for protecting privacy, the control logic 221 can interact with the community layer 205 to anonymize a specific message and corresponding identifiers as described below with respect to the community layer 205").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Muthyala to have the system of claim 28, wherein the AR sports content set includes virtual advertising content positioned within the real-world sports environment as taught by Belimpasakis in order to reduce or eliminates the resource burden (e.g., storage space on the device (See Belimpasakis: Fig. 1, and [0076], "In certain embodiments, the system 100 can initiate or request removal of content associated a bounded area once a node 101 containing the content exits or leaves the bounded area. As noted previously, such content is typically very specific to a location and/or time at the location, and therefore, the content may be of little value when the node 101 is no longer within the bounded area. By initiating or suggesting removal the content, the system 100 advantageously reduces or eliminates the resource burden (e.g., storage space on the device, computing resources associated with storage, etc.) associated with storing information that may no longer be of relevance to a particular node 101"). Muthyala teaches a method and system that may generate AR content using the physical markers arranged in the real word environment to identify and configure marker based interconnectable virtual primitives (building blocks, scene, trees) with rules sets for virtual primitive arrangements; while Belimpasakis teaches a system and method that may generate AR content with contents from the server like the personalized (filtered) advertisements. Therefore, it is obvious to one of ordinary skill in the art to modify Muthyala by Belimpasakis to mix the AR content with filtered advertisements received from the servers. The motivation to modify Muthyala by Belimpasakis is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 50, Muthyala, Dooley, Carignano and Gagner teach all the features with respect to claim 28 as outlined above. Further, Belimpasakis teaches that the system of claim 28, wherein the interaction between the at least two AR object primitives through the at least one primitive interface is via an application programming interface (API) (See Belimpasakis: Fig. 2, and [0047], “The cognition layer 203, together with the community layer 205, provide an application programming interface (API) 225 to enable an application 201 to access the functions of the control logic 221 and the item storage 223. In exemplary embodiments, the API 225 enables application developers to have uniform and easy access to functions related to sharing content or information over the ad-hoc mesh network 109. It is contemplated that the API 225 is extensible to accommodate any application designed to access or use content associated with a bounded area of an ad-hoc mesh network 109. The applications in the various nodes 101 do not have to be the same or mutually compatible. It is sufficient that the applications use the API correctly to be able to publish and search content or information in the surrounding nodes 101”).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights