Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-6, 9, 11, 14-16, 19, are rejected under 35 U.S.C. 103 as being unpatentable over Liu (US 20170287106 A1) in view of Gao (CN 119046192 A).
Regarding claims 11, 1
Liu teaches:
An electronic device, comprising: a graphics processor; and a processor, electrically connected to the graphics processor, (
PNG
media_image1.png
1044
874
media_image1.png
Greyscale
Note: As seen in Liu Fig. 1, an electronic device with a processor unit and a separate graphics processor, GPU 130, that are connected is taught.) configured to receive a plurality of image layers from an application, (Liu Abstract “A device generates blended frames, with each blended frame composed of multiple image layers and each image layer composed of multiple regions. The device includes display hardware. The display hardware retrieves a given image layer in a current frame from a memory.” ¶13 “FIG. 4A illustrates an example of three image layers to be overlaid to form a blended frame according to one embodiment.”
PNG
media_image2.png
480
644
media_image2.png
Greyscale
Note: Liu teaches that a single blended frame is made from collecting and compositing multiple image layers. As seen in Fig. 41, examples of image layers include mobile phone applications, showing one including a search bar, and another application with a search bar. Thus, the ability to receive image layers from applications is taught.) collect processing information, determine whether to enable the local refresh function of the graphics processor; (Liu ¶23 “In one embodiment, the content hint for a region of an image layer may include one or more types, including but not limited to: an alpha hint, a dirtiness hint, a constant hint, and other hints indicating characteristics of the region.” Liu ¶30 “Another type of content hint is the dirtiness hint. When the dirtiness hint of a region of a given image layer indicates the region as being non-dirty at frame N, it means that pixel values of the region of the given image layer for frame N are the same as the pixel values of the region of the given image layer for frame (N−1) (i.e., the previous frame). Thus, for the resulting blended frames stored in memory or used for other processing or analysis (e.g. motion estimation and picture quality processing), the display engine 155 can skip retrieving the non-dirty region of the given image layer if all co-located regions are non-dirty, or a combination of content hints indicates this possibility. When the dirtiness hint of a region of a given image layer indicates the region as being dirty at frame N, it means that at least one pixel value of the region of the given image layer for frame N is different from the pixel value of the region of the given image layer for frame (N−1). Accordingly, the display engine 155 cannot skip retrieving the region of the given image layer for frame N.” Note: Liu teaches that for regions of an image layer, which can be individual pixels or multiple, a dirtiness hint/value is assigned. A “dirty region” or dirtiness value, as referenced by the claims and Liu, refers to a region of the display or layer which is being updated and is not static. Liu teaches that if a region is known to be dirty then it will refresh the dirty region to update the new changes whereas the non-dirty regions will not be updated/refreshed. This teaches the claims language of a “local refresh” as only the local dirty region will receive an update and not the entire image.) wherein the processing information comprises a first image format of an overlapped image output by the graphics processor (Liu ¶19 “The system 100 may include a plurality of image producers, including but not limited to: the processor unit 110, the GPU 130, and the multimedia processing unit 160. The processor unit 110 may generate graphics data to be displayed by the display unit 150, and may also command the GPU 130 to generate graphics data to be displayed.” ¶29 “As mentioned above, one type of content hint is the alpha hint. The alpha hint is a value; e.g., a value of one indicates opaqueness, a value of zero indicates transparency, and a value between zero and one indicates translucency.” ¶21 “Each of the image layers may be divided into a plurality of regions (e.g., tiles), and each region of each image layer includes at least one pixel.” ¶22 “In one embodiment, the content hint for a region of an image layer may include one or more types, including but not limited to: an alpha hint, ” ¶23 “the GPU 130, may generate content hints for the image layers (“first image layers”) that they produce, and store the pixel values of the first image layers” Note: Liu teaches that an “alpha hint” is obtained for every region, where a region can be a single pixel or multiple. An alpha hint is defined as a value from 0 to 1 that defines the opacity for a region/pixel; an alpha hint is simply the alpha channel of a pixel detailing its opacity. Similarly, it is taught that pixel values for image layers are produced and stored by the GPU. This implicitly teaches that the image format is known. For example, if the image format happens to be RGB then the pixel values will represent red, green, and blue channels, without knowing the format the GPU would not be able to produce and store a proper image from said values. Similarly, since the alpha value is obtained and read Liu teaches the system is aware that its image format contains an alpha/opacity channel.) and a dirty region of the image layers; (Liu ¶23, ¶30, cited above, teach that regions of an image layer have dirtiness hint/value assigned to them which specifies whether or not they are a dirty region.) wherein the graphics processor is configured to set an overlay process to be used for the image layers(Liu ¶43 “The dirtiness hints may be generated by software, e.g., the analyzer 210 (FIG. 2) or other image producers, e.g., the GPU … In contrast to the alpha hints generated by the display engine 155, the dirtiness hints for an image layer that is updated at frame N are ready for use at frame N. The Layer Update signal mentioned above in connection with Table 410 only indicates that at least one pixel value in an entire image layer is updated. The dirtiness hint has a finer granularity at the level of a region of an image layer. For example, for Layer-1 at frames 3 and 5, the dirtiness hints indicate that only 3% of the regions in Layer-1 is updated. For Layer-2 at frame 5, the dirtiness hints indicate that only 10% of the regions in Layer-2 is updated … Thus, whenever a region of an image layer is updated, the respective co-located regions in all other image layers need to be retrieved to form a blended frame. For example, at frame 4, Layer-1 has a 3% update and Layer-2 has a 10% update. Each of the three image layers of frame 4 has memory access of 3%+10%=13%, meaning that 13% of the regions in the respective image layers needs memory access.” Note: Liu ¶30, cited previously, teaches that regions that are not updated, aka non-dirty regions, remain the same and are not refreshed, and only dirty regions are updated to produce the overlayed image, aka the frame. Thus, Liu teaches that the overlay process, or the process of generating a new frame, is performed when an identified dirty region needs updating. More info on the process if provided in Liu ¶43, which teaches the precise percentage of image layers which are dirty and require updating are found and updated in the process of forming a “blended frame”, aka the overlayed image. Furthermore, the “dirty hints”, aka the value which indicates if a region is dirty, are taught to be generated by the GPU, which has similarly been shown to generate the image content in Liu ¶19 and ¶23. This shows Liu teaches that this refresh step is a function of the graphics processor. While the graphics processor will update the image layers with the new content it draws and indicate which regions are dirty, it only “sets” the overlay process to occur as it will provide new image layers which require compositing/overlaying to produce the frame. Neither this portion of the claims, nor Liu itself, specifies it is the graphics processor that will carry out the overlay process.) when it is determined to enable the local refresh function of the graphics processor,(Liu ¶23, cited above, specifies the determining process which will determine if image content has changed and dirty regions are present which require updating, and will update only the dirty region image content) wherein the graphics processor is configured to perform the overlay process on the image layers to output the overlapped image with the image format, (Liu ¶43, cited above, teaches the overlay/blending process of image layers is performed to produce the final frame/overlapped image. As no image format conversion is taught the final overlapped image/frame is in the same image format as the image layers. Liu ¶19, ¶29, and ¶21, cited previously, teach that the image format of the layers is known and required for the operations performed.) and refresh the dirty region of the overlapped image. (Liu ¶43, teaches that the specific dirty regions of image layers are updated to produce the frame with the proper updated content, Liu ¶30 further teaches that it is only the dirty regions which are refreshed, and non-dirty regions can efficiently be left the same without respectively fetching their content.)
While Liu teaches that the image format and dirty regions of image layers are identified in order to perform a local refresh operation of the graphics processor to produce an overlapped image it is not taught that when the local refresh function is performed the image format of the overlapped image is changed from the first format to a second format, where the second format is smaller than the first. This is found in Gao which teaches changing the first image format of the overlapped image to a second image format; wherein data size of the second image format is less than that of the first image format; (Gao ¶62, cited below, teaches an overlapped image converted from a first format, ARGB8888, to a smaller second format, RGB888) performing the overlay process on the image layers to output the overlapped image with the second image format, (Gao ¶49 “Please continue to refer to Figure 4, the direct memory access circuit 400 also includes a mixing module 440; the mixing module 440 includes a first input terminal and a second input terminal, the first input terminal receives the first data or the conversion data of the first data, the second input terminal receives the second data or the conversion data of the second data, and the output terminal of the mixing module 440 outputs the mixed third data; the mixing module 440 performs weighted mixing based on the various color channel values and transparency values in the data received by the first input terminal and the second input terminal to determine the third data” ¶50 “After the data is split into a foreground layer and a background layer, the two can be remixed into unified data through a mixing module 440. For example, in an application scenario of photo processing (such as an ID photo), the foreground layer can be a portrait, and the background layer can be a pure white background; … the background layer can also be replaced with a pure blue background, and then the data of the foreground layer and the background layer are weighted and mixed based on the color channel values and transparency values therein to obtain the modified photo.” ¶52 “Therefore, the data format of the first data and the second data is still RGB888. In order to obtain a better mixing effect, the preset data format can be set to ARGB8888, that is, the parameter of increasing transparency. In this way, the input data processing module 430 makes the data format of the conversion data of the first data and the conversion data of the second data both ARGB8888;” ¶62 “The third data is the output data of the mixing module 440, which meets the preset data format, i.e., ARGB8888; however, the storage space occupied by the data format of ARGB8888 is large, and users tend to choose the more commonly used data format of RGB888. In most cases, the effect presented by the image data of RGB888 is sufficient to meet the needs of users. Therefore, the first output data processing module 450 can be used to process the received third data in the data format of ARGB8888 into the fourth data in the data format of RGB888 to meet the needs of users. The output optimization module 460 is configured to receive the fourth data, rearrange the fourth data into the second type of data in units of pixels, and output the second type of data to the output buffer module 470 so that the second type of data matches the storage mode of the output buffer module 470.” Note: Gao teaches a “mixing module 440” that accepts a “first data” and “second data” input and overlays them to produce a final output, which it refers to as the “third data”. The “third data” is the claims overlapped image, an example can be seen in Gao ¶50 where the “first data” is a portrait image, and the “second data” is background image, the mixing module takes into account the opacity values of the alpha channel to mix/overlay the images and produces a final output with the portrait overlayed on the background. Gao ¶52 teaches that images can be better overlayed if they are handled in ARGB, also referred to as RGBA, simply an RGB format with an A alpha/opacity channel, which allows the opacity of certain areas to be taken into account when overlapping. As pointed out by ¶62, ARGB takes up more space than RGB, Gao points out that many users would prefer the smaller RGB format. Due to the efficiency in a smaller file format Gao ¶62 teaches the “third data”, which is the claims overlapped image, is converted from a first larger format, ARGB8888, to a second smaller format, RGB888.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Liu with Gao where an image format and dirty regions of image layers are identified in order to perform a local refresh operation of the graphics processor to produce an overlapped image converts the overlapped image to a second image format that is smaller.
There are several reasons that would motivate one to do so, larger image formats like RGBA contain useful info like opacity when compositing/overlapping or editing the image. This info is not required when displaying a final image, meaning storage space and transfer time can be reduced by using a smaller image format, for example converting from RGBA to RGB, without losing out on detail.
Regarding claims 14, 4,
Liu teaches:
The electronic device as claimed in claim 11, wherein the first image format
While Liu teaches that it is implicitly aware of the image format it works with it does not teach the conversion of image formats from a first format to a second format, where first format is has red, green, blue, and alpha channels, and the second image format is the same without the alpha channel. This specific type of conversion can be found in Gao which teaches wherein the first image format of the overlapped image comprises red, green, blue, and alpha; and the second image format of the overlapped image comprises red, green, and blue. (Gao ¶62, cited previously, teaches that its “third data”, which is the product of mixing, or overlapping, two images together, is converted from ARGB8888, a type of image format with red, green, blue, and an alpha (opacity) channel, to RGB888, a smaller format that does not contain the alpha channel.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Liu with Gao where a final image format of an overlapped image is converted from a first format comprising red, green, blue, and alpha, to a second format that only contains red, green, and blue.
There are several reasons that would motivate one to do so, while the alpha(opacity) information is needed for properly overlaying the images it is unnecessary if no more overlaying is to be performed, one could save on space and file transfer time by simply dropping the alpha channel when the final frame/overlayed image is produced.
Regarding claims 15, 5,
Liu teaches:
The electronic device as claimed in claim 11, further comprising:
While Liu teaches a graphics processor produces the image content, indicates which regions of an image are dirty and in need of updating, produces the pixel values including the alpha values, and will enable the local refresh function, it is not taught that the graphics processor will overlay the images itself to produce the final frame/overlayed image. This is found in Gao which teaches enabling the graphics processor to perform the overlay process on the image layers when the overlay process is set. (Gao ¶50 “After the data is split into a foreground layer and a background layer, the two can be remixed into unified data through a mixing module 440. For example, in an application scenario of photo processing (such as an ID photo), the foreground layer can be a portrait, and the background layer can be a pure white background; … the background layer can also be replaced with a pure blue background, and then the data of the foreground layer and the background layer are weighted and mixed based on the color channel values and transparency values therein to obtain the modified photo.”¶32 “In addition, since the DMA circuit that processes image data has operations such as image format conversion and mixing, and the pipeline level is long,” ¶65 “In the field of image signal processors (ISP), such as in camera modules or other imaging devices, DMA is used to efficiently transfer image data from a sensor to a memory or processor for further image processing, such as face recognition, image enhancement, etc. In the field of display controllers, in a graphics processing unit (GPU) or a display controller, DMA is used to quickly transfer image data to a display device,” Note: Gao teaches that a mixing module which accepts multiple image inputs and overlaps/mixes them, an example is provided where a portrait is put over a background layer. This mixing operation is taught to be handled by a DMA circuit, a direct memory access circuit which efficiently interacts with memory without needing the CPU. Specified with further detail in ¶65, a graphics processor, in this case a GPU, provides the DMA circuit. As the DMA circuit which performs the overlaying process on images happens on the GPU, Gao teaches that a graphics processor is used to perform the overlay process.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Liu with Gao where the overlaying process is performed by the graphics processor.
There are several reasons that would motivate one to do so, as Liu already leverages the graphics processor to produce the image content, indicate which regions of an image are dirty and in need of updating, produce the pixel values including the alpha values, and enable the local refresh function, time could be saved to make the system more efficient by simply not transferring the relevant image layer content out of the GPU and performing the overlay process where all of the image data is already present in the graphics processor.
Regarding claims 16, 6,
Liu teaches:
The electronic device as claimed in claim 11, further comprising:
While Liu teaches the use of a graphics processor it does not teach the use of a direct memory access component used with the graphics processor. This is found in Gao which teaches a memory; a write direct memory access (WDMA) component, electrically connected between the graphics processor and the memory,(Gao ¶32 and 49, cited below, specify that a mixing module, which exists on the DMA circuit, performs the overlay process and will produce an output overlayed image that will be stored in memory. As the DMA(direct memory access) circuit has the ability to produce new image content and store, or write, the content to memory a write direct memory access component is taught.) configured to store the overlapped image with the second image format in the memory; and a read direct memory access(RDMA) component, electrically connected to the memory, configured to read the overlapped image with the second image format from the memory. (Gao ¶61 “the direct memory access circuit 400 also includes a first output data processing module 450, an output optimization module 460, an output cache module 470 and a second output data processing module 480; the first output data processing module 450 is configured to receive the third data and process the third data into fourth data, the third data meets the preset data format, and the fourth data does not meet the preset data format; the output optimization module 460 is configured to receive the fourth data, rearrange the fourth data into second-category data in units of pixels, and output the second-category data to the output cache module 470 so that the second-category data matches the storage mode of the output cache module 470; the output cache module 470 is configured to receive and store the second-category data, and the storage mode of the output cache module 470 matches the width of the system bus” ¶50 “After the data is split into a foreground layer and a background layer, the two can be remixed into unified data through a mixing module 440. For example, in an application scenario of photo processing (such as an ID photo), the foreground layer can be a portrait, and the background layer can be a pure white background; … the background layer can also be replaced with a pure blue background, and then the data of the foreground layer and the background layer are weighted and mixed based on the color channel values and transparency values therein to obtain the modified photo.”¶32 “In addition, since the DMA circuit that processes image data has operations such as image format conversion and mixing, and the pipeline level is long,” ¶65 “In the field of image signal processors (ISP), such as in camera modules or other imaging devices, DMA is used to efficiently transfer image data from a sensor to a memory or processor for further image processing, such as face recognition, image enhancement, etc. In the field of display controllers, in a graphics processing unit (GPU) or a display controller, DMA is used to quickly transfer image data to a display device,” Note: Gao ¶61 teaches that a direct memory access circuit accepts a first and second data. Gao ¶32 teaches that the DMA circuit has a mixing module, Gao ¶50 elaborates on the mixing module further by teaching that it uses the first and second data and overlays them to produce a single image. As the mixing module and terminals that have the fist and second data are all on the single direct memory access circuit, which ¶65 teaches is on the GPU, Gao teaches the ability to read data with its direct memory access circuit, or the claims RDMA. Furthermore, Gao teaches that once the DMA circuit produces the “third data”, the overlayed image, it will convert it to the “fourth data” of a different format. As specified further in the rejection of claim 1, this is Gao converting the image type to the second format. Gao will write, or save, this second format overlayed image to memory, thus teaching the ability to read and write memory with DMA on the graphics processor.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Liu with Gao reading and writing operations pertaining to the overlayed image and other image operations use a read and write direct memory access component that is connected to the graphics processor and memory.
There are several reasons that would motivate one to do so, the invention, Liu, and Gao all detail leveraging the graphics processor for producing image content as is standard practice. Many modern GPUs already contain their own DMA components allowing for read and write operations to be done directly to memory without relying on the CPU, taking advantage of the likely already existing component on the graphics processor provides a more efficient way to handle memory operations.
Regarding claims 19, 9,
Liu teaches:
The electronic device as claimed in claim 15, further comprising: enabling the graphics processor to perform the overlay process on the image layers by setting a layer composition type as a client.
The specifications define the layer composition type of “client” as ¶46 “the method to overlap the image layers of the present invention enables the graphics processor to perform the overlay process on the image layers by setting a layer composition type as a client”, with no other accompanying explanations or information. Based on this, “client” layer composition type is interpreted simply as performing the layer overlay/composition process with the graphics processor. Meaning when interpreted properly, claims 19/9 introduce no new content from the previously rejected claims 15/5 and they are therefore rejected under the same rationale.
Claim 2, 3, 7, 12, 13, 17, are rejected under 35 U.S.C. 103 as being unpatentable over Liu (US 20170287106 A1) in view of Gao (CN 119046192 A) and further in view of Baran (US 20240107031 A1)
Regarding claims 12, 2,
Liu teaches:
The electronic device as claimed in claim 11, wherein the processing information comprises display information (Liu ¶20 “The display unit 150 may include a display engine 155, which is a piece of hardware controlling a driving circuit (not shown) and a display screen (not shown) where frames are to be displayed. The display engine 155 also controls access to the memory unit 140. The display unit 150 may further include a compositor 151. The compositor 151 is a piece of hardware which can be configured to generate a resulting blended frame (also referred to as “frame”) according to images or graphics data, such as a plurality of overlay image layers (also referred to as “image layers”)” ¶21 “ Each of the image layers may be divided into a plurality of regions (e.g., tiles), and each region of each image layer includes at least one pixel.” Note: While the term “display information” is broad and could refer to any number of relevant values like resolution, screen size, display type, etc… claim 3 below and the specifications define it as the display count and size. Display count in the specifications is defined as ¶40 “The display count refers to the number of screens connected to the electronic device. This includes scenarios involving screen mirroring and screen recording, which are considered virtual multi-screen environments.”. Liu teaches the screen count is known, as the device is specified to have a single display with no mentions of screen mirroring or recording. Furthermore, the display size is implicitly known as the display engine, a hardware controller, is responsible for providing the screen with frames to display. It is similarly taught that the display can be divided into a number of pixel regions, which can be single or multiple pixels. For the screen to be divided into individual pixels, and for the hardware controller to provide properly sized frames, the display size must be implicitly known, thus Liu teaches that the display size is known and used for displaying content.) and layer information of the image layers; (Liu ¶21 “Each of the image layers may be divided into a plurality of regions (e.g., tiles), and each region of each image layer includes at least one pixel. The regions of an image layer can be equally-sized or non-equally-sized.” ¶40 “FIG. 4A shows three overlay image layers: Layer-1 (topmost), Layer-2 and Layer-3 (background), which are to be overlaid to form a blended frame. For a sequence of six frames to be used as an example below, 95% of the regions in Layer-1 are transparent regions (indicated by “T” in Table 410 of FIG. 4B), 80% of the regions in Layer-2 are transparent regions, 15% of the regions in Layer-2 are opaque regions (indicated by “O” in Table 410), and Layer-3 has no transparent regions”
PNG
media_image3.png
526
660
media_image3.png
Greyscale
Note: Liu teaches that multiple pieces of information about the layers is known and used. For example, the image layer size can be defined by a plurality of regions, where the regions are a single pixel or group of pixels. This shows that the size of each layer is known as the number of pixels that compose each layer is known. As shown by Liu ¶40 and the accompanying Fig. 4A, the layer order and numbering are also known. As Liu teaches multiple pieces of relevant info about the layers, Liu teaches that layer information is known.) the step of determining whether to enable the local refresh function of the graphics processor, comprises: determining whether the layer information of the image layers of the previous frames has changed; (Liu ¶30 “When the dirtiness hint of a region of a given image layer indicates the region as being non-dirty at frame N, it means that pixel values of the region of the given image layer for frame N are the same as the pixel values of the region of the given image layer for frame (N−1) (i.e., the previous frame).” Liu ¶43, cited previously, teaches that identified dirty regions are updated in the current frame, teaching a local refresh function if the content of the frame, or image layers, has changed in the current frame, . Note: Liu teaches checking if the image layer information from previous frames, in this case the most recent previous frame, is checked against the current frame to determine if the image layer has a dirty region, or in other words if the layer content has changed. These dirty regions will prompt an update, or refresh, to those local regions which is handled by the GPU as seen in ¶43 cited above.)
While Liu teaches checking if the image layer information from previous frames has changed to determine whether or not to do a local refresh it does not teach that the previous 10 frames are checked in determining whether or not to perform a local refresh. Similarly, Liu does not teach checking the dirty region size against a threshold to see if it is below that amount, both of these checks can be found in Baran which teaches the step of determining whether to enable the local refresh function of the graphics processor, comprises: determining whether the layer information of the image layers of the previous 10 frames has changed;(Baran ¶58 “At block 504, the local user device 102 determines whether the image frame (N) includes a change relative to a previous image frame (N−1). For example, the application processing circuitry 207 (FIG. 2 ) can determine whether the image frame (N) includes one or more dirty region(s) based on one or more changes relative to the previous image frame (N−X).” ¶46 “In the example of FIG. 3 , the local user device 102 generates a fifth image frame 328. In this example, the encoder circuitry 230 determines that a size of an encoded version of a dirty region(s) in the fifth image frame 328 and its associated location is greater than a size of the available storage space in the cache 224. Specifically, the storage space in the cache 224 is occupied by the dirty regions 306, 308, 314, 316, 322, 324 of the second, third, and fourth image frames 304, 312, 320 such that there is not enough room to store the dirty region(s) in the fifth image frame 328.” Note: Baran teaches checking previous frame (N-1) for changes in image content which indicate a dirty region. Aside from just checking the most recent last frame, Baran teaches that the (N-X) frame can also be checked, teaching the ability to check against the multiple available previous frames. As seen in Fig. 5 below, dirty regions that are identified from comparing previous frames are encoded, or saved, to the frame. This teaches that the local dirty regions will update, or refresh, the frame, teaching the claims “local refresh” function. In Baran ¶46 an example is provided where 4 previous frames and their dirty regions are stored, and a fifth is unable to be stored due to cache size constraints. As seen in Baran ¶22 and Fig. 2, cited below, the cache size is the cache of programmable circuitry, which could be a device like a CPU or GPU. There are a wide range of available CPUs and GPUs available for use with a variety of cache sizes, meaning the ability to store a previous 10 frames in a cache and check the current against the previous 10 frames, or any number of frames that can fit in a cache being used, is taught.) and determining whether the dirty region occupies a smaller area than a threshold percentage in the image layers.(Baran ¶59 “At block 506, the local user device 102 determines whether a size of the detected dirty region(s) satisfies (e.g., is less than or equal to) a pixel quantity threshold. In some examples, the pixel quantity threshold is based on an available storage space in the cache 224 … ” Abstract “after the size of the dirty region satisfies a threshold: encode the dirty region of the video frame to generate an encoded dirty region; and cause storing of the dirty region in the cache; and after the size of the dirty region does not satisfy the threshold: cause storing of the video frame in a volatile memory that is separate from the cache; and encode the video frame via inter-encoding to generate an encoded video frame.”¶22 “The first programmable circuitry 205 and the second programmable circuitry 208 of FIG. 2 may be instantiated (e.g., creating an instance of, bring into being for any length of time, materialize, implement, etc.) by programmable circuitry such as a Central Processor Unit (CPU) and/or a Graphics Processor Unit (GPU) executing first instructions.”
PNG
media_image4.png
750
538
media_image4.png
Greyscale
PNG
media_image5.png
662
554
media_image5.png
Greyscale
Note: Baran teaches that the size of a dirty region is checked to be smaller than a threshold, where the threshold is a certain size of the screen in pixel quantity. As screens/displays are made up of pixels, to check a threshold below a certain pixel amount is analogous to checking a certain percentage of the screen size. The pixel quantity threshold that the dirty region size is checked against is taught to be determined by available space in the cache 224, which can be seen above in Fig. 2 as a piece of second programmable circuitry. Baran defines this second programmable circuitry which will have the cache 224 in ¶22 as being a type of CPU or GPU. There are a wide range of available CPUs and GPUs with different cache sizes available, thus the specific pixel quantity threshold that is limited by the cache size is a variable amount depending on the hardware. Baran Abstract teaches that when the dirty region is smaller than the threshold, we are able to make the optimization of storing encoding the region to the cache, or in other words we can save only the updated content to the cache instead of going to main memory. Once this is done, we are able to produce an update frame that includes the new updated content contained in the dirty regions, teaching the claims “local refresh function” which locally updates or refreshes the content of a frame with the new content in a dirty region.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Liu with Baran where: determining whether or not to perform a local refresh depends on checking previous frames for changes checks the past 10 previous frames for changes and checks the to be updated dirty regions against a size threshold.
There are several reasons that would motivate one to do so, while information from a current frame from a previous provides a local view of how image content is changing it may be too small of a time frame depending on how fast a system produces frames or outputs them, by checking over a wider span of frames such as 10 frames the described method can be made to work in more situations. Similarly, the local refresh function involves refreshing the dirty regions and not other regions, such a step may be unnecessary and not worth the overhead cost if the dirty region is the entire display, or a large portion of the display. Thus, we can make sure we only optimize when it benefits us by checking if the dirty region is below a certain size threshold.
Regarding claims 13, 3,
Liu teaches:
The electronic device as claimed in claim 12, wherein the display information comprises a display count and a display size;(Liu 20, cited above, teaches that a single display is used in the device described. Liu 21, cited above, teaches that the image layers that make up the frame, which fills the display, is divided into regions made up of single or multiple pixels Note: Display count in the specifications is defined as ¶40 “The display count refers to the number of screens connected to the electronic device. This includes scenarios involving screen mirroring and screen recording, which are considered virtual multi-screen environments.”. As Liu teaches that the display count is known, and is in this case a single screen, and provides no reference to screen mirroring or recording, the display count is known. Liu ¶21 teaches that the image layers which make up the frames, the content of the display, are composed of regions which are single pixels or groups of pixels. As the entire content that makes up the display is defined by its pixel count, Liu teaches that the size of the display is known.) and the layer information of the image layers comprises a layer count, a layer size, (Liu ¶40 “¶40 “FIG. 4A shows three overlay image layers: Layer-1 (topmost), Layer-2 and Layer-3 (background), which are to be overlaid to form a blended frame. For a sequence of six frames to be used as an example below, 95% of the regions in Layer-1 are transparent regions (indicated by “T” in Table 410 of FIG. 4B), 80% of the regions in Layer-2 are transparent regions, 15% of the regions in Layer-2 are opaque regions (indicated by “O” in Table 410), and Layer-3 has no transparent regions”
PNG
media_image3.png
526
660
media_image3.png
Greyscale
”Note: As seen in Fig. 4A Liu knows layer count and position.) the dirty region, ( Liu ¶43, in claim 1, teaches that dirty regions specifically for image layers are identified.) a layer alpha,(Liu ¶22, cited in claim 1, teaches alpha values for image layers are identified and known.) a layer position,(Liu Fig. 4A, cited above, teaches that the layer positions are known, and are denoted by their assigned count number.) and a layer name.(Liu Fig. 4A and ¶40, cited above, teaches that aside from a layer count being taught layers are assigned an identifier, or “name”. In this case, the layer identifier or “name” is their numbered order in the stack, with the first and last layers receiving the name of topmost and background alongside from their numeric identifier.)
Regarding claims 17, 7,
Liu teaches:
The electronic device as claimed in claim 12,
Liu does not however teach that the threshold value that the dirty region size is checked against is 20% of the display size, this is found in Baran wherein the threshold percentage is 20%. (Baran ¶59 “At block 506, the local user device 102 determines whether a size of the detected dirty region(s) satisfies (e.g., is less than or equal to) a pixel quantity threshold. In some examples, the pixel quantity threshold is based on an available storage space in the cache 224 … ” ¶22 “The first programmable circuitry 205 and the second programmable circuitry 208 of FIG. 2 may be instantiated (e.g., creating an instance of, bring into being for any length of time, materialize, implement, etc.) by programmable circuitry such as a Central Processor Unit (CPU) and/or a Graphics Processor Unit (GPU) executing first instructions.”
PNG
media_image4.png
750
538
media_image4.png
Greyscale
Note: Baran teaches that the threshold a dirty region is checked to be smaller than is a pixel amount, where the pixel amount is limited by what can be stored in a cache. The cache is the cache of a “second programmable circuitry”. Baran defines this second programmable circuitry which will have the cache 224 in ¶22 as being a type of CPU or GPU. There are a wide range of available CPUs and GPUs with different cache sizes available, thus the specific pixel quantity threshold that is limited by the cache size is a variable amount. Thus, the ability for the threshold’s pixel amount to be 20% of the screen, or any variable amount of the screen’s size, is taught by Baran and can be determined by selecting a piece of hardware with the desired cache size. It is known that threshold being exactly 20% is not of particular importance, as the specifications ¶41 state “In some embodiments, the threshold percentage may be 20%, but the present invention is not limited thereto”, thus the 20% and any other similar value can be used as a threshold as taught by Baran.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Liu with Baran where the threshold value that is checked is 20%.
There are several reasons that would motivate one to do so, the local refresh function works efficiently by only refreshing the dirty regions which are in need of an update. If the dirty region occupies a large portion of the frame, it may be inefficient to only handle the dirty region as opposed to the entire frame. To avoid an inefficient optimization attempt we can check the that the dirty region is below a certain threshold value, which could be 20% as a range of threshold values is possible.
Claims 8, 10, 18, 20, are rejected under 35 U.S.C. 103 as being unpatentable over Liu (US 20170287106 A1) in view of Gao (CN 119046192 A) in view of Baran (US 20240107031 A1) and further in view of Kwa (US 12094433 B2)
Regarding claims 18, 8,
The electronic device as claimed in claim 12, further comprising: enabling the local refresh function of the graphics processor in response to the layer information of the image layers of the previous 10 frames and the dirty region occupying less than the threshold percentage in the image layers.
The entire content of the claim above is nearly identical to what is contained in claim 7, and is rejected under the same rationale with Baran. While Baran teaches that a variable number of previous frames can be checked, including 10, depending on the available memory, and that the dirty regions is less than a threshold, when Baran checks previous frames, it looks for the presence of a change not a lack of new information. Checking the opposite condition of Baran, that there has been no change in the previous frames is found in Kwa which teaches enabling the local refresh function of the graphics processor (Kwa Col. 16 Line 19 “At 504, the system determines if the current frame needs to be updated. For example, the display engine may send a partial frame update to the TCON. In another example, the display engine may lack the ability to determine what portion of the frame content changes and the display engine sends the full frame update (independent of whether there is a change). TCON can be configured to determine what changes are needed to update the frame (e.g., by comparing a cyclic redundancy check per column between incoming frame data and the frame data that is stored in the local frame buffer, using an XOR of previous pixel content and current pixel content, etc.) and then power manage the column driver(s) accordingly.” Note: It is taught that the system compares a current, new frame data with the previous framed data in memory, in this case the local frame buffer. If new content is identified that is only a small, local part of the frame, and not the entire frame, it is taught that only a partial frame update to the new areas is performed. This action of only refreshing the local area that will be updated in a new frame is analogous to the claims local refresh operation.) in response to the layer information of the image layers of the previous frames not being changed (Kwa Col. 6 Line 31 “If there is no new image data received from the display engine, the display will still refresh at sixty (60) Hz per second because the pixels in the display will decay away if not refreshed. A static image on a display is not really a static image, even though the image is not changing because it is being rewritten or redisplayed sixty (60) times a second for a display with a sixty (60) Hz refresh rate … This can consume a relatively large amount of power so panel self refresh (PSR) was developed to save power for full-screen images. The idea behind PSR is to shut down the display engine and associated circuitry when the image to be displayed on a display is static. More specifically, most current TCONs include a frame buffer and the frame buffer in the TCON can maintain a display image without receiving video image data from the display engine. For a static image, this allows the display engine to enter a low-power state. Allowing the display engine to power down between display updates can save some power and extend the battery life.” Note: Here, Kwa teaches that “if there is no new image data”, an optimization can be performed. Kwa specifically teaches performing the optimization step, in this case shutting down the display engine, when the display is static. Thus, Kwa teaches checking if the display is static, or has no new information in previous frames in order to make an optimization)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Liu with Kwa where checking if the dirty region is below a threshold and checking the previous 10 frames checks the previous frames in order to determine a lack of new content rather than the presence of new content.
There are several reasons that would motivate one to do so, knowing that the image content is not actively changing is relevant for many reasons, the devices computing power and battery life can be optimized by reducing refresh rate, stopping drawing of redundant frames, etc… are all possible optimizations that can make the system more efficient if we are aware that the frame content is not changing.
Regarding claim 20, 10,
The electronic device as claimed in claim 12, wherein the step of refreshing the dirty region of the overlapped image comprises:
Liu does not however teach the two conditions checked by Baran that a previous 10 frames have changed, or that a dirty region is below a threshold. Baran teaches refreshing the dirty region of the overlapped image until the layer information of the image layers of the previous 10 frames has not changed (Baran ¶58 and ¶46, cited in the rejection of claim 2, teach that previous frames are checked to determine if dirty regions have an image update. As specified by Baran Fig. 2 and ¶22, the amount of previous frames that can be checked from memory is variable and depends on the cache size of the device’s GPU or CPU, which could be a wide range of cache sizes, including the ability to store 10 frames.) or the dirty region occupies a larger area than the threshold percentage in the image layers. (
PNG
media_image5.png
662
554
media_image5.png
Greyscale
Baran ¶59, and Fig. 5, cited in the rejection of claim 2, teach checking if the dirty region occupies an area smaller than a threshold, if the check fails, as seen in the flow chart of Fig. 5, we will no longer be able to simply refresh the dirty region of the frames/image layers, and must refresh the entire frame.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Liu with Baran where the checks of checking a previous 10 frames for updated content and checking the dirty region is below a threshold will stop refreshing the dirty region when the checks fail.
There are several reasons that would motivate one to do so, the checks described which determine whether or not to perform a local refresh ensure that the dirty region is a small part of the screen, and that the dirty region is actually dirty and has new updated info. If the dirty region is too large it may be more efficient to refresh the whole frame rather than handling the local dirty region, and if there is not any change, we may not want to locally refresh the dirty region or any region. Therefore, the system can be made more efficient if we only locally refresh when the proper checks pass.
Baran teaches checking for the presence of new, updated frame content which may indicate a dirty region needs refreshing. The opposite check, checking for the lack of updated frame content where the check would fail if new content is found, is taught in Kwa.
refreshing the dirty region of the overlapped image until the layer information of the image layers of the previous frames has changed (Kwa Col. 6 Line 31, cited previously in the rejection of claim 8, teaches that image information is checked for a lack of new information so that an optimization can be made if the screen is static.)
It would have been obvious for a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Liu with Kwa where a local refresh that is performed while the dirty regions is below a threshold and an update across the previous 10 frames is present instead checks for a lack of updates in the previous 10 frames, meaning the condition will be broken if the dirty region exceeds the threshold or an update is identified.
There are several reasons that would motivate one to do so, some optimizations that may make a system more efficient such as reducing the amount of image layers of frames being drawn, reducing frame rate, using a smaller less descriptive image format, are either best performed or can only be performed when image content is not updating. Thus, checking for a lack of updates could allow a number of optimizations to be made depending on the system.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALAN GREGORY HAKALA whose telephone number is (571)272-7863. The examiner can normally be reached 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALAN GREGORY HAKALA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights