Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-5 and 11-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 11, 15-16 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Song et al. (US2021/0227151) in view of Android Developers, Camera2 API Reference: OutputConfiguration (API level 26, corresponding to Android, publicly released Oct 30, 2020, available at web.archive.org/web/20201030040744/https://developer.android.com/reference/android/hardware/camera2/params/OutputConfiguration).
Regarding claim 11, Song teaches a first terminal device (Fig.1-2: First terminal 100) comprising:
a processor, a storage device, and computer programs that are stored in the storage device and can be executed by the processor (See Fig. 1 for processor 170, memory 120 and paragraph 50 for software stored in memory 120), cause the processor to:
store original image data captured by a camera into a first off-screen buffer, and obtain first image data (paragraphs 9, 58, 62 and 74 teaches storing an original image in a buffer. The buffer performs the function of an offscreen buffer because it stores the originally shot frame. Paragraph 16 also teaches that the first terminal includes a plurality of different buffers for the stages);
perform an image rendering processing by invoking the first image data into a rendering process and an encoding process (paragraphs 65, 73-75 teaches performing “collecting/coding/transmitting/displaying” – i.e., both a rendering/display path and an encoding/transmission path. Fig. 5 illustrate the dual-path architecture showing “collection of video frame buffer” feeding into both a “transmission” path (encoding path) and a local display path (rendering path). The rendering and display of the first image is controlled by the preview picture editing module 107 on the first terminal, while a second different version of the same picture/video is sent to the second terminal), obtain a first image and a second image (as discussed, the outputs from the dual-path architecture results in a first image and a second image), the first image representing an image obtained after performing the image rendering processing on the first image data by the rendering process, the second image representing an image obtained after performing the image rendering processing on the first image data by the encoding process, and user operations performed on the first image and the second image being independent (paragraphs 65, 73-75 teaches performing “collecting/coding/transmitting/displaying” – i.e., both a rendering/display path and an encoding/transmission path. Fig. 5 illustrate the dual-path architecture showing “collection of video frame buffer” feeding into both a “transmission” path (encoding path) and a local display path (rendering path). The rendering and display of the first image is controlled by the preview picture editing module 107 on the first terminal, while a second different version of the same picture/video is sent to the second terminal. Additionally, paragraphs 80-86 discloses that a suer may perform picture adjustment operation (translate, zoom, rotate) on the local preview window, and these operations are recorded as “picture adjustment parameters” that are applied separately when generating the final group photo – they are not directly applied to the encoding/transmission path in real time. Furthermore, Song in paragraph 62 discloses that under asynchronous snapping, each group photo party independently controls whether to take a snapshot, confirming that user operations on the preview (first image) are independent from the snapping/encoding action (second image path).);
display the first image on a terminal screen of the first terminal device in a widget preview mode (Fig. 6, show the first terminal displaying the first image in a preview video stream in a designated region of the display interface. Song also teaches in paragraphs 61 and 75 that the preview template management module providing a “segmentation mode of a group photo picture” including “picture-in-picture mode” a recognized widget preview mode);
send the second image to a second terminal device, the second terminal device being a terminal device that initiates an invocation request of a hardware device to the first terminal device (Paragraphs 70-71 and 95 teaches wherein first terminal transmits the encoded second image to the second terminal and that “the second terminal initiates the group photo scenario by accepting or responding to a group photo request, meeting the claimed “invocation request”).
While Song already teaches, at a system level, that its video-frame buffer feeds two distinct downstream operations from the same captured data: a local rendering/display path, "[t]he video communication engine 106 is further configured to perform picture adjustment processing, displaying, and rendering" (¶[0065]) producing the on-screen preview (¶¶[0075], [0079]), and a separate encoding/transmission; "video down-sampling, pre-processing, and video encoding" for network transmission to the other terminal (¶[0074]; Fig. 5). What Song does not expressly disclose is the specific buffer/texture implementation: that the shared buffer is an off-screen buffer bound to an OpenGL ES external ("extended") texture utilized for both processes of rendering and encoding.
Android Camera2 supplies exactly this implementation detail and meets the claimed “wherein the first off-screen buffer is bound to an OpenGl for Embedded Systems (OpenGL ES) extended texture created by the first terminal device, the first off-screen buffer is shared by a rendering process and an encoding process”. Its OutputConfiguration.enableSurfaceSharing() documentation states: "more than one compatible surface can be attached to an OutputConfiguration so that they map to one camera stream, and the outputs share memory buffers when possible," and gives as its principal example "if the camera device uses the same private buffer format between a SurfaceView/SurfaceTexture and a MediaRecorder/MediaCodec, createCaptureSessionByOutputConfigurations will succeed". Which means that one underlying camera buffer shared between a rendering-side consumer (SurfaceTexture/SurfaceView) and an encoding-side consumer (MediaCodec/MediaRecorder). Android Camera2 further discloses that the two consumers act independently on the shared buffer, requiring their own synchronization: "clients should be careful when adding surface outputs that modify their input data... camera clients should have an additional mechanism to synchronize read and write access between individual consumers."
A person of ordinary skill implementing Song's video communication engine, particularly on Android-based commercial hardware would look to the camera framework's own standard facility for exactly this purpose. Android Camera2's OutputConfiguration.enableSurfaceSharing() is feature within Android standards publicly documented platform feature (since Oct 2020) purpose-built to let a single camera buffer be shared, with independent read/write access, between a SurfaceTexture-backed rendering consumer and a MediaCodec-backed encoding consumer. Substituting this known, platform-standard buffer-sharing mechanism for the unspecified buffer arrangement in Song is nothing more than applying a known technique to a known method ready for improvement, to yield the predictable result of avoiding duplicate buffer copies and reducing memory/CPU overhead when one camera capture must simultaneously serve local display and network transmission, which is a benefit of obvious value on resource-constrained commercial/POS-class hardware. See KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398, 416 (2007)
Regarding claim 15, Song teaches the claimed wherein the processor is further caused to: receive an image operation instruction input by a user; execute the image operation instruction by the rendering process, and performing corresponding operations the first image, and obtain a third image; display the third image on the terminal screen of the first terminal device in the widget preview mode (Song in paragraph 80 discloses: "the preview picture editing module 107 is provided for the user to adjust (translate/zoom/rotate) a picture of the user until the user is satisfied." This constitutes receiving an image operation instruction input by a user (translate/zoom/rotate gesture or menu input). Song in paragraph 83 discloses: "the user may adjust the preview picture of the user by using the gesture, including manners such as translating, zooming, rotating, and flipping", the rendering/display process (video communication engine 106 interacting with preview picture editing module 107) executes these operations on the locally displayed preview image. Song further discloses that after adjustment, "the video communication engine sends an adjusted preview picture to each group photo party through a video stream", the adjusted preview image (third image in Claim 5 terms, being the result of applying the user's operation instruction to the first image) is displayed on the terminal screen in the widget preview mode. Song further confirms the adjusted result is displayed in the local preview until the user is satisfied.)
Method claims 1 and 5 are rejected for the same reasons as discussed above in claims 11 and 15, respectively.
Medium claims 16 and 20 are rejected for the same reasons as discussed above in claims 11 and 15, respectively.
Claims 2-4, 12-14 and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Song et al. (US2021/0227151) in view of Android Developers, Camera2 API Reference: OutputConfiguration (API level 26, corresponding to Android, publicly released Oct 30, 2020, available at web.archive.org/web/20201030040744/https://developer.android.com/reference/android/hardware/camera2/params/OutputConfiguration) and further in view of Yu et al. (US2021/0289165).
Regarding claim 12, Song teaches the claimed wherein the processor is further caused to: create a second off-screen buffer in the rendering process; store the first image data into the second off-screen buffer, and obtain second image data; perform an image rendering processing on the second image data by the rendering process, and obtain the first image (Song teaches the claimed in paragraph 0074 wherein “that the camera data is collected into a video frame buffer and subsequently processed through rendering for display.).
While Song teaches the claimed as discussed above, Yu teaches in paragraph 232 that GLSurfaceView rendering is used for local display of camera data obtained via SurfaceTexture, and SurfaceTexture constitutes an off-screen buffer in the rendering pipeline. The creation of second off-screen buffer within the rendering process for staging the image data prior to final rendering is standard known technique Android-based camera display systems as evidenced by Yu’s disclosure.
One of ordinary skill in the art at the time of the filing of the instant application would understand that implementing Song’s rendering path using Yu’s SurfaceTexture/GLSurfaceView involves creating and using an off-screen buffer (second off-screen buffer) within the rendering process for the local preview. One of ordinary skill in the art would have been motivated because combing these references amount to implement a known technique (Yu’s off-screen buffer) in a known system (Song’s dual path video communication) to achieve a predictable result (i.e.- camera image data available to both rendering and encoding paths via an off-screen buffer).
Regarding claim 13, Song teaches the claimed wherein the processor is further caused to: add third image data in the encoding process, the third image data being image data other than image data that collected by the camera of the first terminal device; perform the image rendering processing on the first image data and the third image data by the encoding process, and obtain the second image (Song in paragraphs 97-100 disclose that the group photo generation module 105 obtains high-definition images from multiple group photo participants and composites these with the first terminal’s own camera data to generate the group photo image).
Regarding claim 14, Song teaches the claimed wherein the processor is further caused to: store the third image data into the second off-screen buffer; correspondingly, obtain the first image by performing the image rendering processing on the second image data by the rendering process, comprises: obtaining the first image, by performing the image rendering processing on the second image data and the third image data by the rendering process (Song in paragraphs 75 and 85 teaches that the preview picture editing module 107 renders the local preview including both the local video stream and remote party video streams together in the preview display – i.e. the rendering process operated on both the local camera data and the received remote image data to produce the displayed composite preview. Song in paragraph 75 also teaches that the “first terminal displays, in a default mode in the left half region, a real-time preview video stream (a part of the first video stream) of a shooting scenario (a first scenario) of the first terminal, and displays, in the right half region, a teal-time preview video stream (a part of the second video stream) of a shooting scenario ( a second scenario) of the second terminal” – both are rendered together into the preview display. Storing the third image data (remote party video) into the rendering buffer (second off screen buffer as implemented per Yu’s SurfaceTexture architecture) before jointly rendering both data sets is the standard and natural implementation of Song’s composite preview display, as such compositing requires staging all image data sources in the rendering buffer). The prior motivation as discussed above is incorporated herein.
Method claims 2-4 are rejected for the same reasons as discussed above in claims 12-14, respectively.
Medium claims 17-19 are rejected for the same reasons as discussed above in claims 12-14, respectively.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GELEK W TOPGYAL whose telephone number is (571)272-8891. The examiner can normally be reached M-F (9:30-6 PST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Vaughn can be reached at 571-272-3922. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GELEK W TOPGYAL/Primary Examiner, Art Unit 2481
Prosecution Insights