DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114 was filed in this application after a decision by the Patent Trial and Appeal Board, but before the filing of a Notice of Appeal to the Court of Appeals for the Federal Circuit or the commencement of a civil action. Since this application is eligible for continued examination under 37 CFR 1.114 and the fee set forth in 37 CFR 1.17(e) has been timely paid, the appeal has been withdrawn pursuant to 37 CFR 1.114 and prosecution in this application has been reopened pursuant to 37 CFR 1.114. Applicant’s submission filed on 13 April 2026 has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-2, 5-9, 11-13, 17-18, 21-22, and 24-25 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-2, 5-8, 11-13, and 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hofmann et al. (US 2015/0040074 A1) in view of Rakshit (US 10592199 B2).
Regarding Claims 1, 11, 12, and 21, Hofmann discloses a system performing a method for displaying video of a scene, comprising: capturing, by a user device and at a first time, first image data of the scene, including a first array of pixels representing a first video associated with the scene [Hofmann: ¶ [0021] If desired, higher quality tracking features may be provided by a feature extraction module in a system remote from the user device. In one embodiment, an image frame from the live image data stream is captured in response to receiving the first user input], the user device having an internal tracking system to determine a pose of the user device [Hofmann: ¶ [0100]: Accelerometer 12 may also be used to determine the orientation of user device 1018, such as whether it is being held in portrait mode or landscape mode (i.e., for an elongated device)] and further having a display device [Hofmann: ¶ [0010] An augmented reality system or an augmented reality device may include a display part (e.g., LED screen) that shows the augmented reality space (referred to as "augmented reality view") by combining image frames from an live image data stream from a digital imaging part (e.g., camera) with the augmented reality content]; receiving, by the user device and from a video management system (VMS) [Hofmann: FIG. 6: AR Client 7; and ¶ [0058] A collection of target objects may be maintained and organized based on a fingerprint of an image of the target object in a server remote from the user device. The collection of target objects and their respective fingerprints enable the recognition of objects. To facilitate the management of augmentations associated with a particular target object, the remote server having the fingerprints may provide computer vision processes to enable the recognition of the object. Upon successful recognition of the target object, augmented reality content associated with the target object may then be provided to the user device for display], second image data of the scene, including one or more second arrays of pixels representing one or more second videos associated with the scene [Hofmann: ¶ [0072]: In some embodiments, a reference image is also returned (arrow "HQ features; obj id") such that the reference image is displayed when the user is adding augmented reality content for target object 1016. In general, the reference image may be of better quality than the candidate image captured by digital image part 1002, and sometimes more suitable for augmentation], captured at a second time prior to the first time [Hofmann: ¶ [0073]: In some embodiments, object storage 1010 may be at least partially integrated with system 1014 and database used in maintaining fingerprint DB 132. For instance, object data including fingerprints, tracking resources, reference images, location, object IDs may be maintained together in object storage 1010]; creating, based on the first image data of the scene, respective transformation matrices for the multiple surveillance cameras [Hofmann: ¶ [0084] At a given moment, tracker 1004 maintains a state of the target object(s), wherein the state maintains information such as the current pose(s) of the target object(s) being tracked by tracker 1004. Based on the pose estimation information, a transformation of the augmented reality content may be performed by a graphics engine on the basis of the position and orientation information of the pose information. For example, the transformation may be performed through matrix manipulations on the data of augmented reality content 1040, wherein the matrix manipulations are based on the estimated pose information; while Hofmann may not disclose multiple surveillance cameras, the matrices used in Hofmann would facilitate such transformation processes as claimed], wherein a first transformation matrix of the respective transformation matrices provides a mapping between a first coordinate system of a first camera of the multiple surveillance cameras and a second coordinate system of the user device [Hofmann: ¶ [0113]: One of the ways to render the three-dimensional transformed vector graphic (object) into the augmented reality view is to specify two types of matrices: 1) a modelview matrix and 2) a projection matrix. The modelview matrix contains information about the rotation and translation of the camera relative to the object (transformation parameters obtained from the state of the tracker). On the other hand, because the three-dimensional virtual world is displayed in a two-dimensional display, the projection matrix specifies the projection of three-dimensional world coordinates to two-dimensional image coordinates. Both matrices may be specified as homogeneous 4.times.4 matrices, for instance, the same is used by the rendering framework based on the OpenGL framework]; rendering, by the user device, based on the respective transformation matrices [Hofmann: ¶ [0113]] and the pose of the user device [Hofmann: ¶ [0100]: a graphical user interface may be dynamically generated based at least in part on the tilt measured by the accelerometer (i.e., for determining device orientation), such that three-dimensional graphics may be rendered differently based on the tilt readings (e.g., for a motion sensitive augmented reality game)], at least a portion of the second image data associated with at least one of the one or more second videos to be from a perspective of the user device [Hofmann: ¶ [0023]: In some situations, the graphics object is created first with the non-transformed augmented reality content, and then the graphics object is transformed using the three-dimensional pose information in the tracker part to render the graphics object in perspective with the target object. In some situations, the augmented reality content is transformed first and then a graphics object is created in the three-dimensional environment for rendering and display. The graphics object is rendered for display in the display output, the graphics object appearing in perspective with the target object in the augmented reality view. In some embodiments, the graphics object is referred to as a graphical overlay that is used in combination with images from the live image feed in composing the augmented reality view]; creating, by the user device, composite image data by overlaying the rendered image data of the scene upon the first image data of the scene [Hofmann: ¶ [0023]]; and displaying the composite image data on the display device [Hofmann: ¶ [0102]: Digital imaging part 13 captures images of the real world and provides an live image data streamto which AR client 7 has access. AR client 7 running on user device 1018 is configured to generate an augmented reality view (or camera view) by displaying a graphical overlay in display part 5 over the live image data streamfeed from digital imaging part 13. The graphical overlay is generated from the augmented reality content].
Hofmann may not explicitly disclose second image data captured by multiple surveillance cameras.
However, Rakshit discloses second image data captured by multiple surveillance cameras [Rakshit: Col. 4, ll. 7-23: Multi-video viewing mode or multiple-view video content provides multiple different viewing perspectives—that is, user-selectable options for the angle/direction from which to view a scene. Often times a scene is simultaneously shot with multiple cameras from different camera angles. This approach is used in three-dimensional video production. A result is that a same object will have different depths in the field of view of different perspectives. A particular object in the background from one angle might be in the foreground from another angle. Accordingly, the comparative position and distance of the objects relative to the viewing perspective will change depending on what viewing perspective is selected. Aspects described herein can dynamically change the loudness of the audio portions corresponding to the different objects based on the selected viewing perspective, which corresponds to a specific set of comparative distance and positions for the objects; and Col. 3, ll. 27-32: Software installed on a television or other display system such as a computer system of, or in communication with, a television or other display device for multimedia playback can track the viewing perspective that a user has selected (manually or automatically via a default or other setting) to view the multimedia content].
It would have been obvious to one having ordinary skill in the art before the effective filing date to combine the multi-source imaging of Rakshit with the multi-device display system of Hofmann in order to provide improved image viewing for a user with more input data for image generation, resulting in higher image quality.
Regarding Claims 2 and 13, Hofmann in view of Rakshit discloses all the limitations of Claims 1 and 11, respectively, and is analyzed as previously discussed with respect to those claims.
Furthermore, Hofmann in view of Rakshit discloses wherein the user device includes a depth information sensor configured to obtain depth information of the scene [Hofmann: ¶ [0118]: 3D pose estimator of FIG. 6 determines the modelview matrix H comprising the translation and rotation information needed to display content in perspective with the object ("AR view"). 3D pose estimator of FIG. 6 determines H using the relation x=P*H*X, wherein P is determined on the basis of the camera parameters and wherein H is estimated on the basis of the 2D positions (determined by the 2D correspondence estimator) and P using a non-linear optimization procedure].
Regarding Claim 5, Hofmann in view of Rakshit discloses all the limitations of Claim 1, and is analyzed as previously discussed with respect to that claim.
Furthermore, Hofmann in view of Rakshit discloses wherein the respective transformation matrices for the multiple surveillance cameras are created by the VMS [Hofmann: ¶ [0113]].
Regarding Claim 6, Hofmann in view of Rakshit discloses all the limitations of Claim 1, and is analyzed as previously discussed with respect to that claim.
Furthermore, Hofmann in view of Rakshit discloses further comprising transforming the second image data of the scene from the multiple surveillance cameras on the user device [Hofmann: ¶ [0022] In one embodiment, the user device further includes a tracker part. The tracker part, preferably at least partially implemented on the user device as software, comprises processes for estimating the pose information about the target object using for example an image captured from the live image stream. The tracker enables the generation of matrices that would later be used by a graphics engine to create transformed graphics objects so that augmented reality content appears (even though it is rendered in a two-dimensional space) to have a shape and pose in a three-dimensional virtual world; and ¶ [0083]: Tracker 1004 on user device 1018, using LQ or HQ features packages, the image stream from digital input part 1002 and optionally parameters from digital input part 1002, perform tracking of the target object to estimate the pose of that object].
Regarding Claim 7, Hofmann in view of Rakshit discloses all the limitations of Claim 1, and is analyzed as previously discussed with respect to that claim.
Furthermore, Hofmann in view of Rakshit discloses wherein creating the respective transformation matrices for the multiple surveillance cameras comprises: extracting first landmarks from the first image data to obtain user device landmarks, and extracting second landmarks from image data from each camera of the multiple surveillance cameras to obtain camera landmarks for each camera of the multiple surveillance cameras [Hofmann: ¶ [0143] In operation, feature extractor 340 of FIG. 6 may be used to extract candidate features from frame 104. Using these exemplary feature package 750 and feature 770 as reference features, candidate features extracted by feature extractor 340 may be matched/compared with reference features to determine whether the target object is in the frame (or in view)]; comparing the first landmarks against the second landmarks for each camera of the multiple surveillance cameras to determine matching landmarks for each camera of the multiple surveillance cameras [Hofmann: ¶ [0143]: candidate features… may be matched/compared with reference features]; and using the matching landmarks for each camera of the multiple surveillance cameras to create a transformation matrix for each camera of the multiple surveillance cameras [Hofmann: ¶ [0143]: Successful matches are then provided to two-dimensional correspondence estimator 344 to estimate two-dimensional transformations between the reference features and candidate features].
Regarding Claim 8, Hofmann in view of Rakshit discloses all the limitations of Claim 7, and is analyzed as previously discussed with respect to that claim.
Furthermore, Hofmann in view of Rakshit discloses wherein each transformation matrix of the respective transformation matrices includes at least four points [Hofmann: ¶ [0113]: Both matrices may be specified as homogeneous 4.times.4 matrices, for instance, the same is used by the rendering framework based on the OpenGL framework].
Regarding Claim 25, Hofmann in view of Rakshit discloses all the limitations of Claim 1, and is analyzed as previously discussed with respect to that claim.
Furthermore, Hofmann in view of Rakshit discloses further comprising: streaming, by the VMS, video and the respective transformation matrices to the user device; and using, by the user device, the respective transformation matrices and a current pose of the user device to transform the second image data of the scene to be from the perspective of the user [Hofmann: ¶ [0059] The platform for managing augmented reality content generated by users enables community and social sharing and following of augmented reality content associated with target objects. The platform may provide a collection of tracking resources associated with a group of target objects to the user device such that a user may locally explore, hunt and track for any of the group of target objects in augmented reality view. One skilled in the art would appreciate that further extensions may be implemented to enable a community of users in adding, editing, following, removing and/or viewing augmented reality content associated with target objects using the user device].
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hofmann in view of Rakshit as applied to claim 7 above, and further in view of Samarasekera et al. (US 2008/0167814 A1).
Regarding Claim 9, Hofmann in view of Rakshit disclose(s) all the limitations of Claim 7, and is/are analyzed as previously discussed with respect to that claim.
Furthermore, Hofmann in view of Rakshit discloses wherein using the matching landmarks for each camera of the multiple surveillance cameras to create the transformation matrix for each camera [Bertolami: ¶ [0025]].
Hofmann in view of Rakshit may not explicitly disclose comprises: determining a threshold number of matching landmarks; and populating the transformation matrix with 3D locations from the matching landmarks, the 3D locations being expressed in the first coordinate system of the first camera and in corresponding 3D locations expressed in the second coordinate system of the user device.
However, Samarasekera discloses wherein using the matching landmarks for each camera of the multiple surveillance cameras to create the transformation matrix for each camera comprises: determining a threshold number of matching landmarks [Samarasekera: ¶ [0070]: the landmark matching process may further comprise the use of epipolar geometry constraints to eliminate the false matches. Specifically, from the obtained matches between two images, a fundamental matrix F is first estimated via the robust RANSAC technique described in detail in the '498 application. Subsequently, based on the estimated fundamental matrix F, those matches that produce residuals that are larger than a predefined threshold value are treated as false matches and discarded]; and populating the transformation matrix with 3D locations from the matching landmarks, the 3D locations being expressed in the first coordinate system of the first camera and in corresponding 3D locations expressed in the second coordinate system of the user device [Samarasekera: ¶ [0072]: As a result, the Landmark Database 60 comprises a list of video snapshots of the landmarks extracted from the scene at each location along the route during navigation. Since each landmark is represented by a distinctive HOG descriptor, the combination of HOG descriptors with the spatial configuration data (i.e., 2D and 3D coordinate data) of the landmarks creates a distinctive "landmark snapshot" which serves as the fingerprint of the location and landmark. Each landmark snapshot is composed of the 2D coordinates, 3D coordinates and HOG descriptors of the landmarks, and, optionally, the estimated 3D camera location; and ¶ [0043]].
It would have been obvious to one having ordinary skill in the art before the effective filing date to combine the thresholding of Samarasekera with the processing of Hofmann in view of Rakshit in order to improve computational accuracy without overloading the process.
Claim 17 is/are Hofmann in view of Rakshit as applied to claims 1 and 11 above, and further in view of Anandpura et al. (US 2008/0040766 A1).
Regarding Claim 17, Hofmann in view of Rakshit disclose(s) all the limitations of Claim 1, and is/are analyzed as previously discussed with respect to that claim.
Hofmann in view of Rakshit may not explicitly disclose wherein the first image data of the scene includes prior movements of persons, and wherein displaying the composite image data on the display of the user device includes replaying the movements based on a current perspective of the user device.
However, Anandpura discloses wherein the first image data of the scene includes prior movements of persons, and wherein displaying the composite image data on the display of the user device includes replaying the movements based on a current perspective of the user device [Anandpura: ¶ [0019]: In cases where the display devices are employed in a large stadium or large venue, means for determining the location of the device relative to the playing field or track at the venue can be provided. This would be by GPS or radio triangulation or other means of determining geographic position. Upon the device ascertaining the user's position relevant to the playing field surrounded by viewing seats at the venue, a keypad offering a plurality of viewing angles relevant to their location can be provided for user input to provide video such as instant replay to be displayed on the device at angles relevant to their position from multiple video streams from cameras capturing the playing field or track at different angles at the venue].
It would have been obvious to one having ordinary skill in the art before the effective filing date to combine the replay options of Anandpura with the video management of Hofmann in view of Rakshit in order to improve the views available to a user for replay, improving overall user experience.
Claim(s) 18, 21-22, and 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hofmann in view of Rakshit as applied to claim 7 above, and further in view of Bertolami et al. (US 2010/0287485 A1).
Regarding Claim 18, Hofmann in view of Rakshit discloses all the limitations of Claim 1, and is analyzed as previously discussed with respect to that claim.
Hofmann in view of Rakshit may not explicitly disclose wherein creating the composite image data comprises receiving, by the user device, the second image data of the scene with depth information and the respective transformation matrices for the multiple surveillance cameras, and further comprising mapping, by the user device, the second image data into world coordinates in the second coordinate system of the user device, and using, by the user device, a current pose of the user device to transform the mapped second image data to match a view of the user device.
However, Bertolami discloses wherein creating the composite image data comprises receiving, by the user device, the second image data of the scene with depth information and the respective transformation matrices for the multiple surveillance cameras [Bertolami: ¶ [0063]], and further comprising mapping, by the user device, the second image data into world coordinates in the second coordinate system of the user device [Bertolami: ¶ [0040]: If multiple users are located in the different physical areas, and no mapping or physical area data is available for the physical areas, each user and/or user device may map their respective physical areas independently and employ their own determined coordinate system that may be defined early in the mapping process. Then, the users and/or devices may engage in a negotiation process to arrive at a unified system. The negotiation may determine a unified coordinate system unification based on fiducial markers, recognized features, compared landscape topographies, and/or arbitrarily selected local origins], and using, by the user device, a current pose of the user device to transform the mapped second image data to match a view of the user device [Bertolami: ¶ [0006]: An origin and orientation for a unified coordinate system may be generated using a variety of means. In one embodiment, a user device may determine a proposed origin and orientation for a common coordinate system based on the determined precise location, and transmit this proposed origin and orientation to other user devices and/or to the augmented reality system or application and engage in a negotiation to arrive at a common coordinate system. Alternatively, precise location information may be received from user devices at the augmented reality application or system, which may then determine an origin and orientation and transmit that data to the user devices. In other embodiments, a unified coordinate system may be determined for the virtual space, but not the physical space, based on characteristics of the physical area in which user devices are operating].
It would have been obvious to one having ordinary skill in the art before the effective filing date to combine the coordinate transformation process of Bertolami with the process of Hofmann in view of Rakshit in order to provide improved output quality for the combination of various media.
Regarding Claim 22, Hofmann in view of Rakshit discloses all the limitations of Claim 21, and is analyzed as previously discussed with respect to that claim.
Hofmann in view of Rakshit may not explicitly disclose wherein creating the composite image data comprises receiving, by the user device, the second image data of the scene with depth information and the respective transformation matrices for the multiple surveillance cameras, and further comprising mapping, by the user device, the second image data into world coordinates in the second coordinate system of the user device, and using, by the user device, a current pose of the user device to transform the mapped second image data to match a view of the user device
However, Bertolami discloses wherein a frame of the previously recorded image data includes depth information, the depth information including a range for each pixel or pixel group within an image associated with the frame [Bertolami: ¶ [0050]].
Regarding Claim 24, Hofmann in view of Rakshit discloses all the limitations of Claim 1, and is analyzed as previously discussed with respect to that claim.
Hofmann in view of Rakshit may not explicitly disclose wherein the first transformation matrix includes three-dimensional locations of landmarks extracted from the image data captured by the first camera, the landmarks matching second landmarks extracted from the first image data of the scene, and the three-dimensional locations of the landmarks are expressed in the first coordinate system of the first camera and in corresponding three-dimensional locations expressed in the second coordinate system of the user device.
However, Bertolami discloses wherein the first transformation matrix includes three-dimensional locations of landmarks extracted from the image data captured by the first camera, the landmarks matching second landmarks extracted from the first image data of the scene, and the three-dimensional locations of the landmarks are expressed in the first coordinate system of the first camera and in corresponding three-dimensional locations expressed in the second coordinate system of the user device [Bertolami: ¶ [0025]: Device 220 and/or scene-facing detectors 226a and 226b may capture an image of physical area 230, or otherwise detect objects within physical area 230 and/or derive data from physical area 230. Physical area 230 may have within it landmarks that are detectable, including large landmarks such as tree 231 and tower 232, and/or relatively small and, in some embodiments, numerous landmarks that may be used for location determinations, such as distinctively textured portions of various scene objects or distinct and identifiable attributes of various scene objects. Examples of such detectable portions and/or attributes include the patterning of sections of bark on a tree, intricate metal-work on a tower, portions of the letters on a sign, cracks or other unique physical features of objects, etc. Physical area 230 may be a mapped physical space for which physical information is available. For example, cartography or other physical data may be available for the physical space containing physical area 230 from previous mapping activities. Such activities may have been performed by other users of an augmented reality system or by non-users of such a system. In another embodiment, physical area 230 may be in unmapped physical space for which no cartography or other physical data is available. In many embodiments, other users will not be visible, present, and/or detectable within or proximate to physical area 230. Alternatively, one or more other users, such as user 211, may be proximate to or within physical area 230 and may be detected by scene-facing detectors 226a and 226b].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN R MESSMORE whose telephone number is (571)272-2773. The examiner can normally be reached Monday-Friday 9-5 EST/EDT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chris Kelley can be reached at 571-272-7331. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN R MESSMORE/Primary Examiner, Art Unit 2482
Prosecution Insights