Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Regarding 35 USC § 102/103.
Applicant argues:
Solely for the sake of progressing prosecution, independent claim 1 has been amended to recite, among other features, "receiving camera tracking data from a camera tracker with respect to a camera being operated by a first user physically present in a three-dimensional (3D) physical space, wherein the camera tracker captures tracking images of the camera."
For example, Applicant's specification states that "a camera tracker (112) may be used to track time varying movements of the camera (106) Applicant's Specification, Para. [0030]. The camera tracker "may be used to not only track and generate camera tracking data in connection with cameras present in a 3D physical space but also take pictures or tracking images of persons, devices, or other physical objects present in the 3D physical space. Indeed, the tracking images may be used by a tracking data analyzer (e.g., 208 of FIG. 2A, etc.) as described herein to generate at least a part of the camera tracking data." Id., Para. [0051].
Wang fails to disclose or suggest such features. At most, Wang discloses "the concept of an AR selfie." Wang, Para. [0031]. When "user 100 changes the orientation of mobile device 102 in the real-world (e.g., rotates the view direction of the camera), motion sensors (e.g., accelerometers, gyros) of mobile device 102 sense the change and generate motion data that is used to update virtual background content 104." Id. The "real-world cameras are used to drive the orientation of the virtual camera in the virtual environment." Id.
However, Wang is silent regarding a separate "camera tracker" that "captures tracking images of the camera." Rather, Wang discloses the mobile device 102 sensing its own movement via motion sensors.
Examiner replies that:
Applicant has amended the claims to change the scope since the previous action. The amendment(s) necessitate new ground(s) of rejection and are rejected in detail under the § 102/103 headings below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 16-20, 22, 25-27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang U.S. Patent/PG Publication 20190082118 in view of Weising U.S. Patent/PG Publication 20170084051.
Regarding claim 1:
(Currently amended) A method, comprising: (Wang [0081] FIG. 11 illustrates a device architecture for implementing the features and process described in reference to FIGS. 1-10, according to an embodiment. Architecture 1100 can include memory interface 1102, one or more data processors, video processors, co-processors, image processors and/or other processors 1104, and peripherals interface 1106. Memory interface 1102, one or more processors 1104 and/or peripherals interface 1106 can be separate components or can be integrated in one or more integrated circuits. The various components in architecture 1100 can be coupled by one or more communication buses or signal lines.)
receiving camera tracking data from a camera tracker with respect to a camera being operated by a first user physically present in a three-dimensional (3D) physical space (Wang [0034] FIG. 2D illustrates mobile device 102 with viewport 202 and forward-facing camera 204. A viewing coordinate system (X.sub.c, Y.sub.c, Z.sub.c) is shown where the +Z.sub.c coordinate is the forward-facing camera's view direction. In computer graphics, a camera analogy is used where viewer 206 located at a view reference point (VRP) observes a virtual environment through virtual camera 205 and can look and move around the virtual environment. This is accomplished by defining a viewing coordinate system (VCS) which has the position and orientation of virtual camera 205, as shown in FIGS. 2D and 2E.)(Wang [0038] In an embodiment, the location of the virtual camera, in addition to its orientation, can be changed in the virtual environment. For example, the location of the virtual camera can be changed by physically moving the mobile device or by using an GUI affordance (a virtual navigation button). In the former, location data (e.g., GNSS data) and/or inertial sensor data (e.g., accelerometer data) can be used to determine the position of the virtual camera in the virtual environment. In an embodiment, the virtual environment can be 3D video, 3D 360° video or 3D computer-generated imagery (CGI) that can respond to a user's actions.).
generating an image portion depicting one or more visual objects not physically present in the 3D physical space using a camera perspective derived from the camera tracking data (Wang [0042] FIGS. 3C and 3D illustrate graphical user interfaces with different background scenes selected and showing a recording view and full-screen playback view, according to an embodiment. In FIG. 3C, a recording view is shown where user 302c has selected a virtual background 303c. Note that during recording, viewport 301 is not full-screen to provide room for recording controls. In FIG. 3D, a full-screen playback view includes scene selector 313 that can be displayed when user 302d has selected the “SCENES” affordance 312. In an embodiment, scene selector 313 is a touch control that can be swiped by user 302d to select virtual background 303d, which in this example is a Japanese tea garden. Also note that virtual background 303d is now displayed full-screen in viewport 311. [0043] FIGS. 3E and 3F illustrate graphical user interfaces for recording and playing back selfies using a backward-facing camera and showing a recording view and full-screen playback view, according to an embodiment. In FIG. 3E, a recording view is shown with virtual background 303e. Virtual background 303e is what a user would see in front of them through the backward-facing camera in the virtual environment.).
causing the one or more visual objects to be visually combined with the camera perspective into a personal image taken by the first user using the camera, wherein the personal image visually depicts one or more visual objects physically present in the 3D physical space (Wang [0012] FIGS. 3A and 3B illustrate a graphical user interface for recording AR selfies using a forward-facing camera, according to an embodiment. [0013] FIGS. 3C and 3D illustrate graphical user interfaces with different background scenes selected and showing a recording view and full-screen playback view, according to an embodiment.)(Wang [0047] FIG. 5 illustrates compositing layers used in an AR selfie, according to an embodiment. In an embodiment, alpha compositing is used to combine/blend the video data containing an image of the selfie subject with the virtual background content.).
Wang does not teach an outside-in tracker. In a related field of endeavor, Weising teaches:
receiving camera tracking data from a camera tracker with respect to a camera being operated by a first user physically present in a three-dimensional (3D) physical space; wherein the camera tracker captures tracking images of the camera (Weising [0070] In another embodiment, a depth camera in one portable device is used to measure the distance to the other device. Once the distance is known, the reference point can be set based on the position of either device. Additional data, such as image data, can also be used to complement the calculations of the relative positions of the devices.)(Weising [0091] During operations 1702, 1704, and 1706, the first device can exchange game and position tracking information with the second device. The second device performs a similar method to the one performed by the first device in operations 1712, 1714, 1716, 1718, and 1720.)(Weising [0088] By using a combination of motion tracking, image analysis, and high persistence of information between each device, the play area appears in a stable position even if when devices move around.)(Weising [0094] Additionally, the Position Module can analyze sound or image data captured with the cameras and the microphone to calculate the position. Further yet, the Position Module can perform tests to determine the position of the portable device or the position of other devices in the vicinity, such as WiFi ping test or ultrasound tests.)
generating an image portion depicting one or more visual objects not physically present in the 3D physical space using a camera perspective derived from the camera tracking data (Weising [0089] The method flows to operation 1704 where synchronization information data is exchanged with the second device in order to identify a reference point in the 3D space. The reference point is relative to the physical locations of the first and second devices in the 3D space. Additionally, both devices establish the physical location in the 3D space of the other device when setting the reference point.)(Weising [0090] From operation 1704, the method flows to operation 1706 for generating in the display a view of an interactive scene that includes the reference point and one or more virtual objects. The view shows the perspective of the interactive scene as observed from the current location of the device. After the view is shown in the display, a check is performed in operation 1708 to determine if the first device has moved. If the first device has moved, the method flows to operation 1710 to change the perspective for the display according to the new location of the first device. If the device has not moved, the method loops back to operation 1706 to continue updating the display, such as to reflect changes of virtual objects during game play.)
causing the one or more visual objects to be visually combined with the camera perspective into a personal image taken by the first user using the camera, wherein the personal image visually depicts one or more visual objects physically present in the 3D physical space (Weising [0040] FIG. 2 illustrates a virtual reality scene observed with the portable device. After synchronizing device 104 with respect to reference points 106, the portable device will start displaying a view of the virtual reality 108. The view in the display is created by simulating that a camera in the back of the portable device moves within the 3D space around reference point 106. FIG. 2 depicts a virtual reality that includes a chess board.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use an outside-in tracker as taught by Weising. The motivation for doing so would have been to provide a synchronized shared-space experience (Weising [0085]). Therefore it would have been obvious to combine Weising with Wang to obtain the invention.
Regarding claim 2:
The method of Claim 1, has all of its limitations taught by Wang in view of Weising. Wang further teaches wherein the camera represents one of one or more cameras included in a user device (Wang [0044] FIG. 4 is a block a diagram of system 400 illustrating the processing steps used in the creation of an AR selfie, according to an embodiment. System 400 can be implemented in software and hardware. Forward-facing camera 401 generates RGB video and IR depth sensor 402 generates depth data, which are received by Audio/Visual (A/V) processing module).
Regarding claim 16:
(New) The method of Claim 1, has all of its limitations taught by Wang in view of Weising. Wang further teaches wherein the one or more visual objects are combined into the personal image taken by the camera by way of contemporaneously rendering the image portion on a screen image display in the 3D physical space (Wang [0047] FIG. 5 illustrates compositing layers used in an AR selfie, according to an embodiment. In an embodiment, alpha compositing is used to combine/blend the video data containing an image of the selfie subject with the virtual background content.)(Wang [0073] Process 900 can continue by rendering for display composite media (e.g., a composite video) in a viewport of the mobile device (907). During a recording operation, the composite media is presented as a live video feed. When the user changes the view direction of the real-world camera, the virtual camera transform updates in real-time the virtual background content in sync with the real-world camera. The recorded AR selfie video can be played back from storage through the viewport and also shared with others on, for example, on social networks.).
Regarding claim 17:
(New) The method of Claim 1, has all of its limitations taught by Wang in view of Weising. Wang further teaches wherein the one or more visual objects are combined into the personal image by way of superimposing the image portion onto a pre-combined personal image taken by the camera (Wang [0047] FIG. 5 illustrates compositing layers used in an AR selfie, according to an embodiment. In an embodiment, alpha compositing is used to combine/blend the video data containing an image of the selfie subject with the virtual background content.)
Regarding claim 18:
(New) The method of Claim 17, has all of its limitations taught by Wang in view of Weising. Wang does not teach simulating lighting although they teach light information (Wang [0082] Sensors, devices and subsystems can be coupled to peripherals interface 1106 to facilitate multiple functionalities. For example, one or more motion sensors 1110, light sensor 1112 and proximity sensor 1114 can be coupled to peripherals interface 1106 to facilitate motion sensing (e.g., acceleration, rotation rates), lighting and proximity functions of the mobile device.). In a related field of endeavor, Weising teaches:
wherein lighting conditions existing in the 3D physical space are simulated in the image portion superimposed onto the pre-combined personal image (Weising [0084] The virtual scene can be made even more realistic by using shadows and lighting determined by the lighting sources in the room. By using camera feeds, game environments and characters have scene lighting and shadows influenced by the real world. This means that a player's hand will cast a shadow over virtual characters or objects as the hand are reaches into the virtual world to interact with the virtual objects. Game world shadows and lighting are adjusted by real world shadows and lighting to get the best effect possible.).
Therefore, it would have been obvious before the effective filing date of the claimed invention to match lighting as taught by Weising. The motivation for doing so would have been improved realism (Weising [0084]). Therefore it would have been obvious to combine Weising with Wang to obtain the invention.
Regarding claim 19:
(New) The method of Claim 1, has all of its limitations taught by Wang in view of Weising. Wang further teaches wherein a depth map is received and used to cause at least one of the one or more visual objects to visually appear, in the personal image, to have physical contact with a visual object physically present in the 3D physical space (Wang [0050] Referring to FIGS. 6A and 6B, matte generation process 600 can be divided into three stages: preprocessing stage 603, RGB-D matting stage 604 and post-processing stage 605. Process 600 takes as input RGB video data 601 that includes images of the subject and a depth map 602 that includes the depth data provided by the IR depth sensor. It should be observed that depth map 602 includes areas of shadow where the depth data is undefined. Note that the shadow along the left contour of the subject's face is thicker (more undefined data) than along the right contour of the subject's face. This is due to the offset between the IR projector and the IR camera. Each of stages 603-605 will be described in turn below.)(Wang [0046] With virtual camera and position information, the user can walk around the 3D scene with 3D characters.)
Regarding claim 20:
(New) The method of Claim 1, has all of its limitations taught by Wang in view of Weising. Wang further teaches wherein the camera perspective is represented by a spatial position and a spatial orientation, of the camera wherein the spatial position and the spatial orientation, of the camera, are determined in reference to the 3D physical space (Wang [0034] FIG. 2D illustrates mobile device 102 with viewport 202 and forward-facing camera 204. A viewing coordinate system (X.sub.c, Y.sub.c, Z.sub.c) is shown where the +Z.sub.c coordinate is the forward-facing camera's view direction. In computer graphics, a camera analogy is used where viewer 206 located at a view reference point (VRP) observes a virtual environment through virtual camera 205 and can look and move around the virtual environment. This is accomplished by defining a viewing coordinate system (VCS) which has the position and orientation of virtual camera 205, as shown in FIGS. 2D and 2E.)(Wang [0038] In an embodiment, the location of the virtual camera, in addition to its orientation, can be changed in the virtual environment. For example, the location of the virtual camera can be changed by physically moving the mobile device or by using an GUI affordance (a virtual navigation button). In the former, location data (e.g., GNSS data) and/or inertial sensor data (e.g., accelerometer data) can be used to determine the position of the virtual camera in the virtual environment. In an embodiment, the virtual environment can be 3D video, 3D 360° video or 3D computer-generated imagery (CGI) that can respond to a user's actions.)(Wang [0082] Sensors, devices and subsystems can be coupled to peripherals interface 1106 to facilitate multiple functionalities. For example, one or more motion sensors 1110, light sensor 1112 and proximity sensor 1114 can be coupled to peripherals interface 1106 to facilitate motion sensing (e.g., acceleration, rotation rates), lighting and proximity functions of the mobile device. Location processor 1115 can be connected to peripherals interface 1106 to provide geopositioning and process sensor measurements. In some implementations, location processor 1115 can be a GNSS receiver, such as a Global Positioning System (GPS) receiver chip. Electronic magnetometer 1116 (e.g., an integrated circuit chip) can also be connected to peripherals interface 1106 to provide data that can be used to determine the direction of magnetic North. Electronic magnetometer 1116 can provide data to an electronic compass application. Motion sensor(s) 1110 can include one or more accelerometers and/or gyros configured to determine change of speed and direction of movement of the mobile device. Barometer 1117 can be configured to measure atmospheric pressure around the mobile device.).
Regarding claim 22:
(New) The method of Claim 1, has all of its limitations taught by Wang in view of Weising. Wang further teaches further comprising:
receiving second camera tracking data with respect to a second camera being operated by a second user physically present in the 3D physical space generating a second image portion depicting one or more second visual objects not physically present in the 3D physical space using a second camera perspective derived from the second camera tracking data causing the one or more second visual objects to be visually combined with the second camera perspective into a second personal image taken by the second user using the second camera. (Wang multiple users [0103]-[0105])(See claim 1)
There is a prima facie case of obviousness since the limitation is directed to common practices which the court has held normally require only ordinary skill in the art and hence are considered routine expedients are discussed below. See MPEP 2144.04. Duplication of Parts “the court held that mere duplication of parts has no patentable significance unless a new and unexpected result is produced.” The rationale to modify is that it is obvious to try, since Wang has multiple users, and there could be no users, one user, and multiple users, where there is a reasonable expectation of success since the devices are entirely separate and the users are merely operating the systems near each other.
Regarding claim 25:
(New) The method of Claim 1, has all of its limitations taught by Wang in view of Weising. Wang further teaches wherein the image portion is generated using one of:
3Drendering models, single view images, multi view images, or stereoscopic images (Wang [0025] In an embodiment, a selfie subject can be composited with virtual background content extracted from a virtual environment data model. The virtual background content can include but is not limited to: a two-dimensional (2D) image, a three-dimensional (3D) image and 360° video.).
Regarding claim 26:
The claim is a parallel version of claim 1. As such it is rejected under the same teachings.
Regarding claim 27:
The claim is a parallel version of claim 1. As such it is rejected under the same teachings.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang U.S. Patent/PG Publication 20190082118 in view of Weising U.S. Patent/PG Publication 20170084051 and Gray U.S. Patent/PG Publication 9911237.
Regarding claim 21:
(New) The method of Claim 1, has all of its limitations taught by Wang in view of Weising. Wang in view of Weising does not teach qr codes. In a related field of endeavor, Gray teaches:
wherein at least a part of the camera tracking data is received by tracking and decoding a visual representation of a quick response (QR) code presented on a device image display (Gray C7 L55-C8 L5 In embodiments, a flash pattern can comprise displaying or projecting a patterned image that can be detected in the captured image. For example, a mobile device can be equipped with a display on the same surface as the image capture element. For example, the primary display of a mobile device can be on the same surface as a front facing camera. In embodiments, a rear facing camera can be on the rear of the mobile device, and a secondary display can be placed on the rear of the device as well. In embodiments, the image capture element can capture an image that includes a display. The display can include, for example, a QR code, bar code, or another distinguishable pattern. In embodiments, a mobile device can be equipped with a projector, which can project a distinguishable pattern on a surface, even the user, captured in the image.)(Gray C13 L6-C14 L5 Other positioning elements may include QR codes, barcodes, RFID tags, NFC tags, etc. that enable the device to detect and receive location information or identifiers that enable the device to obtain the location information (e.g., by mapping the identifiers to a corresponding location). Various embodiments can include one or more such elements in any appropriate combination.).
Therefore, it would have been obvious before the effective filing date of the claimed invention to use qr codes as taught by Gray . The rationale for doing so would have been that it combines prior art elements according to known methods to yield predictable results since Wang tracks the position of a camera to display AR objects, and Gray tracks the position of a camera to display AR objects, where there are predictable results since the output is the same with tracking and outputting AR objects. Therefore it would have been obvious to combine Gray with Wang in view of Weising to obtain the invention.
Claim(s) 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang U.S. Patent/PG Publication 20190082118 in view of Weising U.S. Patent/PG Publication 20170084051 and Berme U.S. Patent/PG Publication 10231662.
Regarding claim 23:
(New) The method of Claim 1, has all of its limitations taught by Wang in view of Weising. Wang in view of Weising does not teach machine learning. In a related field of endeavor, Berme teaches:
wherein at least a part of the camera tracking data is predicted by a machine learning model that has been trained with training data that includes training tracking images of user devices that include cameras (Berme C43 L20-30 Furthermore, in one or more embodiments, the data acquisition/data processing device 104 that is operatively coupled to the head-mounted visual display device 344 may execute a machine learning algorithm for predictive tracking of the subject's head movement so as to predict how the subject is going to move and pre-render the correct image for that view, thereby significantly decreasing the display latency or display time lag.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use machine learning as taught by Berme. The motivation for doing so would have been decreasing the display latency or display time lag (Berme C43 L20-30). Therefore it would have been obvious to combine Berme with Wang in view of Weising to obtain the invention.
Claim(s) 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang U.S. Patent/PG Publication 20190082118 in view of Weising U.S. Patent/PG Publication 20170084051 and Appleboim U.S. Patent/PG Publication 20210090209.
Regarding claim 24:
(New) The method of Claim 1, has all of its limitations taught by Wang in view of Weising. Wang in view of Weising does not teach qr codes. In a related field of endeavor, Appleboim teaches:
wherein a visual representation of a specific quick response (QR) code is accessible to a user device that includes the camera to establish a data communication link with a system that generates the image portion (Appleboim [0098] Embodiments of the disclosed subject matter may include a system, device, method and computer program product that may allow a user to submit one or more images of himself or herself in order to virtually try-on countless products sold online and/or sold offline, and to see different looks of such products (e.g., clothes, accessories, or the like) shown in a realistic manner on the user's own image. The user may also virtually dress himself or herself by scanning physical codes, such as QR-codes, barcodes, price tags, clothes tags, or the like, captured from physical clothes located in real-life shops; in order to similarly achieve a virtual dressing of the user's image with such items or clothes.)(Appleboim Fig. 1)(Appleboim [0085] In an embodiment, an indication of a product may be received, such as a clothes item, that the user would like to introduce into the anchor image for the purpose of generating a composite anchor image that includes an image of the product. For example, the user may wish to virtually dress an anthropomorphic entity depicted in the anchor image. The product indication may be provided by the user, who may select the item from any suitable source, including but not limited to selecting from an online catalog (webpage), or scanning a barcode or a QR code or a price-tag or a clothes-tag of an actual product in a store, or capturing an image of the product (e.g., from the actual product or from images thereof in print media such as a newspaper or magazine), or entering or dictating or saying the name of a product or vendor, for example, “Old Navy, skirt, model Barbara,” which the may be looked-up online or at a vendor's website in order to extract therefrom the items' data and image. Additional product sources include “inspirational” sources, such as images depicting celebrities wearing identifiable clothing items and/or accessories.).
Therefore, it would have been obvious before the effective filing date of the claimed invention to use qr codes as taught by Appleboim. The motivation for doing so would have been to provide easy access to a greater repository of virtual objects for display in the system (Appleboim [0101]), giving the user more options and customization. Therefore it would have been obvious to combine Appleboim with Wang in view of Weising to obtain the invention.
Conclusion
For the prior art referenced and the prior art considered pertinent to Applicant’s disclosure but not relied upon, see PTO-892 “Notice of References Cited”.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON PRINGLE-PARKER whose telephone number is (571) 272-5690 and e-mail is jason.pringle-parker@uspto.gov. The examiner can normally be reached on 8:30am-5:00pm est Monday-Friday. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, seehttp://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON A PRINGLE-PARKER/
Primary Examiner, Art Unit 2617
Prosecution Insights