Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Priority
Acknowledgment is made of applicant’s foreign priority claim, for U.S. Application No. 19/006,916, based on a foreign application filed on 08/18/2022.
Status of Claims
Claims 1–7 are pending in the application. Claims 1-7 are rejected.
Overview of Grounds of Rejection
Ground of Rejection
Claim(s)
Statute(s)
Reference(s)
Ground 1
1, 3, 5
§ 103
Iyer et al. (US20200241634A1), Krishnaprasad et al. (US20200234395A1), Kim et al. (US20070086665A1)
Ground 2
2, 6, 7
§ 103
Iyer et al. (US20200241634A1), Krishnaprasad et al. (US20200234395A1), Kim et al. (US20070086665A1), Masule et al. (WO2021252178A1)
Ground 3
4
§ 103
Iyer et al. (US20200241634A1), Krishnaprasad et al. (US20200234395A1), Kim et al. (US20070086665A1), Saif et al. (NPL)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
(Please see the cited paragraphs, sections, pages, or surrounding text in the references for the paraphrased content.)
Ground of Rejection 1
Claims 1, 3, 5 are rejected under 35 U.S.C. § 103 as being unpatentable over Iyer et al. (US20200241634A1) in view of Krishnaprasad et al. (US20200234395A1), and further in view of Kim et al. (US20070086665A1).
As per Claim 1, Iyer teaches the following portion of Claim 1, which recites: “A system comprising: a mobile device including:” Iyer et al.’s HMD 102 corresponds to the claimed mobile device because it is worn by the user during execution of the xR application and includes cameras, sensors, and communication interfaces for exchanging data with the host IHS. Iyer et al., ¶¶ [0024], [0035], [0037].
Iyer teaches the following portion of Claim 1, which recites: “a transmitter configured to transmit an image captured by a camera and an inertial measurement unit (IMU) data corresponding to position and posture information via the camera,”
Iyer et al. teaches that “frames captured by camera(s) 108” are “transmitted to IHS 103 via control interface 208”. Iyer et al. also teaches that an “Inertial Measurement Unit (IMU) built into HMD 102” provides angular velocity and accelerometer data used to produce a new “HMD position and/or pose estimation.” Thus, Iyer et al. teaches transmitting a camera image and IMU data corresponding to position/posture information. Iyer et al., ¶¶ [0035], [0053].
Iyer alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Kim et al., they collectively teach some of the limitation(s).
Iyer and Kim teach the following portion of Claim 1, which recites: “the transmitter further configured to combine the image and IMU data into a single image data packet prior to transmission,”
Iyer et al. teaches transmitting camera and sensor data from HMD 102 to IHS 103, but does not clearly teach combining the image and IMU data into a single image data packet. Kim et al. teaches the known packaging technique of encoding metadata “together with media data” to create a “single integrated MAF file”, and also teaches that a single-track MAF is composed of “one media content and final metadata corresponding to the media content.” Kim et al. further teaches transmitting the created MAF file through a communication channel. Applying Kim’s packaging approach to Iyer would have resulted in combining the camera image data with corresponding IMU/pose metadata into one transmitted data unit. Kim et al., ¶¶ [0056], [0118], [0137].
Iyer teaches the following portion of Claim 1, which recites: “and a receiver configured to receive and play back an augmented reality (AR) content at least;”
Iyer et al. teaches that “IHS 103 transmits encoded packetized rendered content” to HMD 102, and that HMD 102 displays an xR view including “virtual objects composited with the projected view of the user's real environment.” This corresponds to receiving and playing back AR content. Iyer et al., ¶¶ [0038], [0089].
Iyer and Kim alone do not explicitly teach all the limitation(s) of the claim. However, when combined with Krishnaprasad, they collectively teach all of the limitation(s).
Krishnaprasad teaches the following portion of Claim 1, which recites: “and a split rendering server configured to render a three-dimensional object using the image and the IMU data of the image data packet of the mobile device, and provide the AR content, which is the rendered result, to the mobile device.”
Krishnaprasad et al. teaches the split rendering server aspect because it states that “the rendering workload can be split between a server and a user device” and that this is “split rendering.” Krishnaprasad et al. further teaches an edge/cloud server for XR/AR/VR split rendering and a device-MEC server architecture. Krishnaprasad et al., ¶¶ [0042], [0045], [0053].
Iyer et al. teaches the rendering operation using camera and IMU/pose data. In particular, Iyer et al. teaches that rendering engine 406 renders an xR model based on the user’s coordinate location and “the user's pose (e.g., from an IMU)”, and that IHS 103, as the server, uses its GPU to render xR content including “3D graphics” for display by HMD 102, with the rendering process receiving SLAM/camera-frame input from the HMD. Iyer et al. further teaches transmitting the rendered packetized content back to HMD 102. Iyer et al., ¶¶ [0070], [0080], [0089].
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to combine Iyer et al.’s camera/IMU-based xR rendering system with Krishnaprasad et al.’s split-rendering server architecture to offload graphics processing from the mobile/HMD device to a server, thereby improving rendering performance and reducing device-side computational burden. A POSITA would further have been motivated to use Kim et al.’s known technique of packaging media data with corresponding metadata into a single integrated transmitted unit so that image data and associated pose/IMU information remain synchronized for server-side rendering. The combination uses known components according to their established functions and would have yielded predictable results, namely server-side rendering of AR/3D content using synchronized camera and IMU/pose data and transmission of the rendered AR content back to the mobile device.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 3, Iyer alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Kim et al., they collectively teach all of the limitation(s).
Kim teaches Claim 3, which recites: “The system of claim 1, wherein the image data packet is configured to be packaged in a packaging format of the ISO/IEC 23000 standard.”
Kim et al. (US20070086665A1) teaches packaging media data/metadata using an MPEG-A multimedia application format, including “MPEG Photo Player Application Format (ISO/IEC 23000-3)” and creating a “single integrated MAF file” by encoding media data together with metadata. Kim further teaches the MAF file structure including a header, MPEG metadata, and media data. Thus, Kim teaches packaging the image/media data packet in an ISO/IEC 23000 MAF packaging format. Kim et al., ¶¶ [0005]-[0006], [0056], [0118]-[0122].
The rationale and motivation to combine the references as set forth for claim 1 are incorporated herein by reference for the present claim.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Iyer and Kim teach Claim 5, which recites: “The system of claim 1, wherein the AR content is configured to include one frame in one image track.”
Iyer et al. teaches AR/xR rendered content as frames, including “virtual objects composited with the projected view of the user's real environment,” server-rendered “3D graphics with left (L) and right (R) views,” and encoded “2D video frames” transmitted to the HMD. Iyer et al., ¶¶ [0038], [0080]-[0082], [0089]. Kim et al. teaches the track structure, disclosing a “single track MAF” composed of “one media content and final metadata corresponding to the media content,” including media data in the track. Kim et al., ¶¶ [0118], [0122]. Thus, Iyer’s AR frame content packaged using Kim’s single-track media structure teaches one frame in one image track.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Ground of Rejection 2
Claims 2, 6, 7 are rejected under 35 U.S.C. § 103 as being unpatentable over Iyer et al. (US20200241634A1) in view of Krishnaprasad et al. (US20200234395A1), further in view of Kim et al. (US20070086665A1), and still further in view of Masule et al. (WO2021252178A1).
As per Claim 2, Iyer alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Masule, they collectively teach some of the limitation(s).
Masule teaches Claim 2, which recites: “The system of claim 1, wherein the image data packet is configured to be packaged in units of five image frames.”
Masule et al. (WO2021252178A1) teaches video-frame packetization, including generating “a set of video packets carrying the video frames” and “packetize the encoded video frames.” Masule further teaches batching frame subsets and states that “a batch (e.g., or subset of frames) includes 5 frames” and “the first five frames.” Thus, Masule teaches packaging image/video frame data in units of five image frames. Masule et al., ¶¶ [0048], [0061], [0063].
Before the effective filing date of the claimed invention, a POSITA would have been motivated to modify the claim 1 combination with Masule’s five-frame batching/packetization to improve video throughput, reduce processing overhead/latency, and provide predictable frame-batch handling for image data packets.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 6, Iyer alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Kim and Masule, they collectively teach all of the limitation(s).
Iyer teaches following portion of Claim 6, which recites: “The system of claim 1, wherein the transmitter includes: the camera configured to capture the image;”
Iyer et al. teaches HMD/mobile-side components that transmit camera and sensor data to the host IHS, including control interface 208 and processor 205 for communication. Iyer et al., ¶¶ [0034]-[0035], [0089]. Iyer teaches camera(s) 108 and that “frames captured by camera(s) 108” are transmitted to IHS 103. Iyer et al., ¶ [0035].
Iyer teaches following portion of Claim 6, which recites: “an IMU data detector configured to detect the IMU data;” Iyer teaches an “Inertial Measurement Unit (IMU) built into HMD 102” that receives angular velocity and accelerometer data and produces “HMD position and/or pose estimation.” Iyer et al., ¶ [0053].
Iyer and Masule teach following portion of Claim 6, which recites: “a stream processor configured to convert the image and the IMU data into a stream data in accordance with synchronization;” Iyer teaches transmitting camera/sensor data, including IR/NIR frame data and SLAM data, while Masule teaches a “video data stream” and synchronization using “temporal information” and “frame timestamps.” Iyer et al., ¶¶ [0056], [0089]; Masule et al., ¶¶ [0055], [0087].
Masule and Kim teach following portion of Claim 6, which recites: “a capturing module configured to split the stream data into five frames and package the image and the IMU data into the single image data packet.” Masule teaches batching video frames and states that “a batch (e.g., or subset of frames) includes 5 frames” and “the first five frames.” Masule et al., ¶ [0063]. Kim teaches packaging media data with corresponding metadata by encoding metadata “together with media data” to create a “single integrated MAF file.” Kim et al., ¶ [0056].
Before the effective filing date of the claimed invention, a POSITA would have been motivated to modify Iyer’s HMD camera/IMU xR data transmission with Masule’s synchronized video-stream batching and five-frame packet handling, and Kim’s media-plus-metadata packaging, to improve throughput, reduce latency/processing overhead, and keep image data synchronized with associated pose/IMU data. The combination applies known streaming, batching, and packaging techniques to Iyer’s known xR transmission system and would have yielded predictable results.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
As per Claim 7, Iyer alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Kim and Krishnaprasad, they collectively teach all of the limitation(s).
Iyer, Kim, and Krishnaprasad teach Claim 7, which recites: “The system of claim 6, wherein the receiver includes: an AR content processor configured to play back or store the AR content according to a setting; a play-back configured to play back the AR content; and a memory configured to store the AR content.”
Iyer et al. teaches AR/xR playback, where HMD 102 is configured to “render and display frames to provide an xR view” including “virtual objects composited with the projected view of the user's real environment.” Iyer et al., ¶ [0038]. Krishnaprasad et al. teaches the receiver-side components: content encoder/decoder 122 and display processor 127 process graphical/AR content, displays 131 “display or otherwise present frames,” and system memory 124 is configured to “store received encoded or decoded graphical content.” Krishnaprasad et al., ¶¶ [0026]-[0029]. Kim et al. further teaches storing media and providing it for browsing/sharing “according to a request from the user,” corresponding to playback/storage according to a setting. Kim et al., ¶¶ [0135]-[0136].
The rationale and motivation to combine the references as set forth for claim 1 are incorporated herein by reference for the present claim.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Ground of Rejection 3
Claim 4 is rejected under 35 U.S.C. § 103 as being unpatentable over Iyer et al. (US20200241634A1) in view of Krishnaprasad et al. (US20200234395A1), further in view of Kim et al. (US20070086665A1), and still further in view of Saif et al. (NPL).
As per Claim 4, Iyer alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Saif, they collectively teach some of the limitation(s).
Saif teaches Claim 4, which recites: “The system of claim 1, wherein the image data packet is configured to be transmitted according to an HTTP QUIC protocol.”
Saif et al. (NPL) teaches HTTP over QUIC, stating that “QUIC … was designed and tuned to carry HTTP traffic, forming the basis of HTTP/3 (H3)” and proposing an “HTTP/3 (H3) solution” using “QUIC transport.” Saif further teaches H3 pub-sub transmission where the client “includes data in message body” for publishing. Thus, Saif teaches transmitting the image data packet according to an HTTP/QUIC protocol. Saif et al. (NPL), Abstract, Sections I and III, Table I, pp. 1-2.
Before the effective filing date of the claimed invention, a POSITA would have been motivated to modify Iyer et al.’s xR system, in which HMD 102 transmits camera/sensor data to IHS 103 and receives rendered xR content, with Saif et al.’s HTTP/3 over QUIC transmission protocol to improve low-latency, interoperable, and timely packet-based communication between the mobile device and server. The combination would have merely used a known HTTP/QUIC transport protocol for Iyer’s known xR data transmissions and would have yielded predictable results.
PNG
media_image1.png
13
460
media_image1.png
Greyscale
Conclusion
The prior art made of record and relied upon in this action is as follows:
Patent Literature:
Krishnaprasad et al. (US20200234395A1) — “Methods and apparatus for standardized apis for split rendering.”
Iyer et al. (US20200241634A1) — “ENCODING CONTENT FOR VIRTUAL, AUGMENTED, AND MIXED REALITY (xR) APPLICATIONS IN CONNECTIVITY-CONSTRAINED ENVIRONMENTS.”
Kim et al. (US20070086665A1) — “Method and apparatus for encoding multimedia contents and method and system for applying encoded multimedia contents.”
Masule et al. (WO2021252178A1) — “Video throughput improvement using long term referencing, deep learning, and load balancing.”
Non-Patent Literature (NPL):
Saif et al. — “A Pure HTTP/3 Alternative to MQTT-over-QUIC in Resource-Constrained IoT”, 2021-10-27. Available at: [https://arxiv.org/pdf/2106.12684]
Note: A PDF copy of each NPL reference is attached with this Office Action. URLs are included for applicant convenience. If a link becomes unavailable in the future, the citation information may be used to locate the reference or access archived versions via the Wayback Machine.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure and is listed as follows:
Patent Literature:
(none)
Non-Patent Literature (NPL):
(none)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADEEL BASHIR whose telephone number is (571) 270-0440. The examiner can normally be reached Monday-Thursday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on (571) 276-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ADEEL BASHIR/
Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights