DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims the benefit of US Provisional Application No. 63/539,633, filed 09/21/2023. Claims 1-20 have been afforded the benefit of this filing date.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
The following terms in the claims have been given the following interpretations in light of the specification:
“at least one framebuffer… is a Direct Framebuffer” (claims 2, 9, and 16):
[0020] “The Screen Buffer may pass the graphical information into a Direct Framebuffer process (e.g., referred to as DirectFB such as DirectFB 124 of process 104). The DirectFB process may be instantiated from a DirectFB library, which may be an open-sources component of the operating system of the display device (e.g., such as a Linux-based operating system, etc.). The DirectFB library provides graphics rendering services to applications with graphics output including native applications, operating system applications or processes, third-party applications, etc.”;
[0021] “DirectFB (e.g., a contraction of Direct Framebuffer) is library that provides a hardware-accelerated graphics environment for embedded systems and other low-level platforms. The DirectFB library enables applications to access and manipulate the graphics hardware directly, without using a traditional windowing system, which may be a fast, efficient way to render graphics on resource-constrained systems.”
Therefore, a “Direct Framebuffer” is not a type of framebuffer as claims 2, 9, and 16 seem to suggest, but a particular library for implementing graphics and display functionality including frame buffers, windowing, and other related functions. Consequently, the limitation in claims 2, 9, and 16 is interpreted as meaning “using the DirectFB library”. This definition is used for purposes of searching for prior art, but cannot be incorporated into the claims.
Should applicant wish different definitions, Applicant should point to the portions of the specification that clearly show a different definition.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 5-9, 12-16, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Marz et al. ("Reducing Event Latency and Power Consumption in Mobile Devices by Using a Kernel-Level Display Server". IEEE Transactions on Mobile Computing, vol. 18, no. 5 (1 May 2019), pp. 1174-1187. https://doi.org/10.1109/TMC.2018.2857809, hereinafter "Marz") in view of He et al. ("Optimizing Smartphone Power Consumption through Dynamic Resolution Scaling". Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom '15) (07 Sep 2015), pp. 27-39. https://doi.org/10.1145/2789168.2790117, hereinafter "He").
Regarding claim 1, Marz teaches: A method comprising:
intercepting a function call from a set of framebuffers to a hardware abstraction layer (pg. 1181 section 5.3 “Linking Android with the KDS”: “The KDS modifies this rendering process in three ways: It intercepts the application’s surface request to SurfaceFlinger, gets a surface from SurfaceFlinger and returns it to the application”, where the “surface” is a framebuffer that the application will draw to – this is further explained by the teachings of He; fig. 7a and 7b show that the SurfaceFlinger communicates with the HAL, and that elements of the KDS are inserted in the section of the pipeline between the application and the HAL), wherein the function call from each framebuffer of the set of framebuffers is associated with a graphical component to be displayed within a graphical user interface (pg. 1181 section 5.3 “Linking Android with the KDS” further explains that the surface associated with the intercepted request is drawn on by the application, a plurality of surfaces are composited together, and the flattened surface is rendered to a mobile display, which can be considered a “graphical user interface”);
compositing the graphical component of one or more framebuffers of the set of framebuffers into a single graphics plane (pg. 1180 col. 1 section 4.2 “The Compositor System”: “The KDS compositor is rather simple in its design since most GUI graphics packages, such as Android’s Surface Flinger handle much of the compositing. However, the difference is that the KDS’ compositor flattens the entire screen, including all GUI attachments, whereas Surface Flinger composes the graphics for each running application.”); and
transmitting the single graphics plane to the hardware abstraction layer for rendering within a display of display device (fig. 4 “The KDS 2-dimensional drawing system draws a composed and flattened surface to the framebuffer of the device that is passed in as a parameter.”; fig. 7(b) shows that kds_draw_2d outputs to the hardware abstraction layer.).
Marz does not explicitly teach: wherein the function call of the one or more framebuffers is intercepted over a time interval.
He teaches that the contents of each application framebuffer are requested by the SurfaceFlinger for composition once every frame (pg. 30 section 3.1 “Android Graphics Overview”: “When a VSYNC signal arrives, SurfaceFlinger collects all the graphics buffers for visible layers and asks the Hardware Composer to composite all visible layers together.”). Marz teaches that the KDS intercepts the communication between the application and the SurfaceFlinger; therefore, in order to render each frame separately, the interception must also occur over a time interval corresponding to one frame.
Additionally, He explicitly clarifies that the set of framebuffers may include multiple framebuffers, each of which is associated with a graphical component to be displayed within a graphical user interface (fig. 4, two applications each have their own buffer for graphical output, where the buffers in BufferQueue 1 and 2 correspond to the “surface” terminology used by Marz).
Marz and He are both analogous to the claimed invention because they are in the same field of improving the performance of a graphical display system and pertain to many of the same issues, including compositing application buffers into a single graphical output and transmitting that output to a hardware interface, and intercepting/overriding standard components of the system. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Marz with the teachings of He. The motivation may have been to further reduce power consumption, as is one of the stated goals of both inventions; also, the cited elements of He are standard features of the Android graphics system that necessarily also apply to the Android-based invention of Marz.
Regarding claim 2, the combination of Marz in view of He teaches: The method of claim 1, wherein at least one framebuffer of the set of framebuffers is a Direct Framebuffer (Marz pg. 1180 section 4.3 “The Drawing System”: “The KDS uses many of the DirectFB routines that are already written in the Linux/Android kernel. As previously mentioned, DirectFB provides helper functions to draw to the framebuffer using the hardware to improve the drawing speed. This allowed us to implement the KDS without having to duplicate DirectFB’s functionality.”).
Regarding claim 5, the combination of Marz in view of He teaches: The method of claim 1, wherein intercepting the function call from the set of framebuffers includes de-registering the function call to the hardware abstraction layer and redirecting the function call to a consolidated compositing block (Marz pg. 1181 section 5.3 “Linking Android with the KDS”: “The KDS modifies this rendering process in three ways: It intercepts the application’s surface request to SurfaceFlinger, gets a surface from SurfaceFlinger and returns it to the application.”;
the KDS, which receives the intercepted request, can be considered a “consolidated compositing block”: pg. 1175 “The KDS also combines compositing and drawing into a single layer in the kernel, which essentially collapses the last three layers of the existing Android rendering pipeline into a single layer.”).
Regarding claim 6, the combination of Marz in view of He teaches: The method of claim 1, wherein each framebuffer is associated with a different process executing on the display device and configured to output a graphical component (He fig. 4, each application has its own buffer for graphical output;
pg. 29-30 section 3.1 “Android Graphics Overview”: “Applications call the OpenGL ES API [31] to use the GPU for graphics processing and to draw their User Interface (UI). The processing result is put into a graphics buffer in a structure called BufferQueue. Each application has its own BufferQueue that has three graphics buffers by default. The system UI process also has a dedicated BufferQueue for drawing the system UI elements such as the “navigation bar” at the bottom of the screen and the “status bar” at the top of the screen.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the invention of Marz in view of He with the cited teachings of He because they are standard features of the Android graphics system that also apply to the Android-based invention of Marz.
Regarding claim 7, the combination of Marz in view of He teaches: The method of claim 1, wherein the function call from a set of framebuffers is intercepted after a windowing process (Marz fig. 7b teaches that the interception occurs between the application and SurfaceFlinger; He pg. 30 section 3.1 “Android Graphics Overview” teaches that “SurfaceFlinger knows the layout of all the UI windows in the system by working with the WindowManager service.”; thus the windowing process of the WindowManager happens before SurfaceFlinger obtains each application buffer for compositing) and before graphics rendering (Marz fig. 7b teaches that the rendering step “kds_draw_2d” happens after the interception; also see pg. 1181 section 5.3 “Linking Android with the KDS”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the invention of Marz in view of He with the cited teachings of He because they are standard features of the Android graphics system that also apply to the Android-based invention of Marz.
Claim(s) 3, 10, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Marz ("Reducing Event Latency and Power Consumption in Mobile Devices by Using a Kernel-Level Display Server") in view of He ("Optimizing Smartphone Power Consumption through Dynamic Resolution Scaling") as applied to claims 1, 8, and 15 above, and further in view of "Module gop" (Docs.rs. Retrieved from Wayback Machine, 07 Feb 2023: https://web.archive.org/web/20230207211531/https://docs.rs/uefi/latest/uefi/proto/console/gop/index.html, hereinafter "GOP").
Regarding claim 3, the combination of Marz in view of He teaches: The method of claim 1, wherein the single graphics plane is transmitted to the hardware abstraction layer (Marz fig. 4 “The KDS 2-dimensional drawing system draws a composed and flattened surface to the framebuffer of the device that is passed in as a parameter.”; fig. 7(b) shows that kds_draw_2d outputs to the hardware abstraction layer.).
The combination of Marz in view of He does not explicitly teach: wherein the single graphics plane is transmitted to a graphics output protocol buffer of the hardware abstraction layer.
GOP teaches: wherein the single graphics plane is transmitted to a graphics output protocol buffer of the hardware abstraction layer (“The UEFI GOP is meant to replace existing VGA hardware interfaces. It can be used in the boot environment as well as at runtime, until a high-performance driver is loaded by the OS. The GOP provides access to a hardware frame buffer and allows UEFI apps to draw directly to the graphics output device.”).
GOP is analogous to the claimed invention because it is in the same field of processing graphical output. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the invention of Marz in view of He with the teachings of GOP to use a standardized output according to UEFI standards. The motivation would have been compatibility with UEFI systems, such as Windows.
Claim(s) 4, 11, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Marz ("Reducing Event Latency and Power Consumption in Mobile Devices by Using a Kernel-Level Display Server") in view of He ("Optimizing Smartphone Power Consumption through Dynamic Resolution Scaling") as applied to claims 1, 8, and 15 above, and further in view of Arm ("AFBC". Retrieved from Wayback Machine, 16 Jun 2023: https://web.archive.org/web/20230616105216/https://developer.arm.com/Architectures/Arm%20Frame%20Buffer%20Compression, hereinafter "AFBC").
Regarding claim 4, the combination of Marz in view of He teaches: The method of claim 1, but does not explicitly teach: wherein the one or more framebuffers are configured to use Arm Framebuffer Compression.
AFBC teaches that the Arm Frame Buffer Compression protocol is useful for improving speed and power efficiency in mobile devices (“The Arm Frame Buffer Compression (AFBC) protocol addresses the difficulty of creating increasingly more complex designs within the thermal limit of a mobile device… In such cases it reduces the overall system level bandwidth and power cost of transferring spatially coordinated image data throughout the system by up to 50%.”), and is applicable to video compression (“One of the most bandwidth intensive use cases is video post processing.”)
The invention of Marz is explicitly designed “to conserve power and reduce latency in mobile devices” (pg. 1183 section 6 “Experimental Results”), and it handles video compression (pg. 1179 section 4.1 “KDS Thread System”: “… (c) a background thread for handling constant activity, such as decompressing video frames…”)
Additionally, AFBC is analogous to the claimed invention because it is in the same field of processing graphical output.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the invention of Marz in view of He with the teachings of AFBC to use the Arm Frame Buffer Compression protocol in order to help further reduce power consumption and latency.
Regarding claims 8-14, they are rejected using the same references, rationale, and motivation to combine as claims 1-7 respectively because their limitations substantially correspond to the limitations of claims 1-7, with the additional limitations of:
A system comprising:
one or more processors;
a non-transitory computer-readable medium storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations (Marz and He each teach methods explicitly designed as software running on a standard mobile device; one of ordinary skill in the art would recognize that this indicates a processor and non-transitory storage).
Regarding claims 15-20, they are rejected using the same references, rationale, and motivation to combine as claims 1-6 respectively because their limitations substantially correspond to the limitations of claims 1-6, with the additional limitation of:
A non-transitory computer-readable medium storing instructions that when executed by one or more processors, cause the one or more processors to perform operations (Marz and He each teach methods explicitly designed as software running on a standard mobile device; one of ordinary skill in the art would recognize that this indicates a processor and non-transitory storage).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Adiletta et al. (US 20200326955 A1) elaborates on various relevant elements of the standard Android graphics pipeline, including compositing buffers associated with each application and transmitting the result to a hardware abstraction layer, and the placement of the window manager before the compositor/SurfaceFlinger.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN STATZ whose telephone number is (571)272-6654. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BENJAMIN TOM STATZ/ Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/ Supervisory Patent Examiner, Art Unit 2611