DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on March 2nd 2023, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response Amendments/Arguments
Applicant’s Amendment filed on November 18, 2025, has been entered and made of record. Claims 1,11, and 19 have been amended, claim 14 has been canceled. Accordingly, Clams 1-13 and 15-21 remain pending.
Response to Amendments/Arguments
Applicant’s arguments, as set forth in the Remark on page 8 filed November 18, 2025, with respect to the rejection(s) of claim(s) 1, 11, and 19 have also been fully considered in view of the amendment to the claims and are found persuasive. Accordingly, the rejection has been withdrawn. However, upon further consideration, a new ground of rejection is made in view of Avila et al (US 2022/0374490 A1).
Avila teaches the use of an Alpha channel for a purpose other than transparency, namely, to store a numerical value associated with each pixel. Specifically, Avila discloses storing a number of accumulated samples per pixel over time in the alpha channel. As described in paragraph [0178] of Avila.
“In this way, some implementations use the alpha channel (in texture) to store the number of accumulated samples per pixel. Each pixel has a different number of accumulated (alpha) values over time. This information is used to improve..”
Accordingly, Avila demonstrates that the alpha channel may be used to store per-pixel numerical information unrelated to transparency. Therefore, using alpha to store and compare numeric sequence information would have been obvious to one of ordinary skill in the art.
Applicant is advised to consider incorporating additional language into the claims to further distinguish over the prior art. For example, the claims may recite operation including “sampling alpha-channel values from a plurality of selected pixel locations of the output frame; alpha channel is a unsigned integer or a floating-point component;comparing the sampled alpha-channel values to an expected sequence number; and, when the comparison indicates a mismatch beyond a configured tolerance, performing a safety action comprising at least preventing presentation of the output frame on a display device.” Please see areas within the enabling specification (see paragraphs 0041, 0049, 00223) that disclose the particulars of these suggestions.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status
The following is a quotation of 35 U.S.C. 103 which forms the basis for all
obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention
is not identically disclosed as set forth in section 102, if the differences between the claimed
invention and the prior art are such that the claimed invention as a whole would have been obvious
before the effective filing date of the claimed invention to a person having ordinary skill in the art
to which the claimed invention pertains. Patentability shall not be negated by the manner in which
the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35
U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness
Claim(s) 1-2, 10-12,17 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over CHANG H Y. (WO 2023283540 A1) hereafter “Chang”, and in view of Jeong et al. US 2023/0237941 A1 hereinafter “Jeong” further in view of Schnebly; Dexter A. US-20090147861-A1 hereinafter “Schnebly” further in view of Avila et al (US 20220374490 A1) hereinafter “Avila”.
Regarding claim 1 Chang-Jeong-Schnebly-Avila
Chang discloses Claim 1. (Currently Amended) A method, comprising:
Determining an expected sequence number corresponding to an output frame (Chang, [0015] “…, the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from the different image sensors, and in turn, produce corresponding output frames” additionally [0014] refers to the image signal processors (ISP) receiving a sequence of image frames and producing a flow of output frames for further processing. See also paragraph [0016]), the output frame generated from a plurality of input frames corresponding to one or more views and captured using one or more cameras ( Chang, [0015] “After an output frame representing the scene is determined by the image signal processor using the image correction, …the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from the different image sensors, and in turn, produce corresponding output frames of image data” [0049] “In some aspects, depth sensors such as depth sensor 140 may be coupled to the image signal processor 112 and output from the depth sensors processed in a similar manner to that of image sensors 101 and 102 to generate corrected image frames based on an image frame captured by the depth sensor 140.” i.e., it discloses input frame depth and image sensors processed similarly, which are derived from different optical path thus constituting different ‘views’ captured using multiple cameras.)
Chang does not explicitly disclose
wherein consecutive input frames captured by the one or more cameras are associated with different sequence numbers;
performing, for each of one or more pixels included in the output frame, a comparison of the expected sequence number with a pixel value stored in [[a]] an alpha pixel channel associated with the pixel, wherein a position of at least one of the one or more pixels corresponds to a position within a display device that is visible to a user; and
based at least on the comparison, preventing the output frame from being presented on the display device.
However, in the same field of endeavor Jeong discloses more explicitly the following:
wherein consecutive input frames captured by the one or more cameras are associated with different sequence numbers; (Jeong, [0113] “...the detecting device 52 receives a plurality of images frames from the generating device 50.” [0115] “…comparing the drive pattern of at least one pixel with a preset drive pattern. Additionally, paragraph, [0116]-[0117] The determining unit checks whether each frame’s drive pattern matches the preset pattern.”
performing, for each of one or more pixels included in the output frame, a comparison of the expected sequence number with a pixel value . . . , . . . ; (Jeong, [0115] “The determining unit 522 determines the occurrence of a screen freeze error by comparing the drive pattern of at least one pixel with a preset drive pattern. ” [0116] “When the drive pattern of the at least one pixel corresponds to the preset drive pattern, the determining unit 522 may determine that no screen freeze error has occurred.”, Paragraph [0120], [0124] go further to describe sampling multiple pixels, storing actual drive pattern data, and comparing it to expected patterns.) and
based at least on the comparison, preventing the output frame from being presented on the display device. (Jeong, [0117] “…when the drive pattern of the at least one pixel does not correspond to the preset drive pattern, the determining unit 522 may determine that a screen freeze error has occurred.” [0121] discloses that if a mismatch is detected between the preset drive pattern and the actual drive pattern of as pixel, a screen freeze error is determined, and the system may perform a screen blanking operation to prevent the faulty frame being shown. Further [0124] Discusses generating display control signals based on the comparison results.)
Therefore, it would have been obvious to a person having ordinary skill in the art before
the effective filing date of the application to modify the teachings of Chang with Jeong to create the system of Chang as outlined above in order to “wherein consecutive input frames captured by the one or more cameras are associated with different sequence numbers; performing, for each of one or more pixels included in the output frame, a comparison of the expected sequence number with a value stored in a pixel channel associated with the pixel. one or more-pixel values of the output frame; and based at least on the comparison, preventing the output frame from being presented on a display device.” as suggested by Jeong
The reasoning is that to “ improve detection accuracy of a screen freeze error and reliability of information transmitted to an occupant in the vehicle.” (Jeong, [0188])
Chang-Jeong does not explicitly disclose
wherein a position of at least one of the one or more pixels corresponds to a position within a display device that is visible to a user;
. . . stored in [[a]]an alpha pixel channel associated with the pixel,
However, in the same field of endeavor Schnebly discloses more explicitly the following:
wherein a position of at least one of the one or more pixels corresponds to a position within a display device that is visible to a user; (Schnebly,[0006] “The code can count, or otherwise change, from one frame to the next. Verification at the destination, or display, of the changing code within the frames of the video stream confirms that the video stream is not in a fault condition.” [0008] “The sequence of video frames with the embedded code values is transmitted to a destination where the sequence of code values is extracted. A second sequence of code values is generated at the destination and then compared with the extracted code values. A fault indication is presented when the comparison does not match and the video frames are displayed when the comparison does match.”, See also Abstract)
Therefore, it would have been obvious to a person having ordinary skill in the art before
the effective filing date of the application to modify the teachings of Chang-Jeong with Schnebly to create the system of Chang-Jeong as outlined above, “wherein a position of at least one of the one or more pixels corresponds to a position within a display device that is visible to a user;” as suggested by Schnebly. In particularly, one of ordinary skill in the art would have been motivated to combine the teaching of Chang-Jeong with Schnebly in order to archive the benefit that “Extracting and verifying a sequential code can be a much more efficient operation than calculating a checksum or CRC over each frame of a video stream” (Schnebly, [0006])
Chang-Jeong-Schnebly does not explicitly disclose
. . . stored in [[a]]an alpha pixel channel associated with the pixel,
However, in the same field of endeavor Avila discloses more explicitly the following:
. . . stored in [[a]]an alpha pixel channel associated with the pixel, (Avila, [0178] “In this way, some implementations use the alpha channel (in texture) to store the number of accumulated samples per pixel. Each pixel has a different number of accumulated (alpha) values over time. This information is used to improve..”)
Therefore, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the application to modify the teachings of Chang-Jeong and Schnebly in view of Avila to store a value in an alpha pixel channel associated with the pixel, as taught by Avila.
One of ordinary skill in the art would have been motivated to incorporate Avila’s use of the alpha channel into the system of Chang-Jeong and Schnebly in order to improve real-time image rendering by enabling storage and use of per-pixel numerical information without introducing additional data structure. (Avila, [0178])
Note: The motivation that was utilized in the rejection of claim 1, applies equally as well to claims 2, 10-12,17 and 19.
Regarding claim 2, Chang-Jeong-Schnebly-Avila
Chang-Jeong-Schnebly-Avila discloses 2. (Original) The method of claim 1, further comprising:
performing a second comparison of a second sequence number with one or more additional pixel values of a second output frame (Jeong, [0133] “Based on the comparison result of the comparing unit 524, the monitoring unit 520 may monitor a drive pattern of at least one pixel in the porch area. [0136] “The first image frame 60 includes a first back porch area 600, a first active area 602, a first front porch area 604-, and first-pixel data 601. The second image frame 62 includes a second back porch area 620, a second active area 622, a second front porch area 624, and second-pixel data 621. The third image frame 64 includes a third back porch area 640, a third active area 642, a third front porch area 644-, and third-pixel data 641.” See Fig. 5); and
based at least on the second comparison, causing a and presentation of the second output frame on the display device (Jeong, see FIG. 8, ¶ [0154] Jeong discloses, “The display panel may sequentially output the first image frame 70, the second image frame 72, the third image frame 74, and the fourth image frame 76.”).
Regarding claim 10 Chang-Jeong-Schnebly-Avila
Chang-Jeong-Schnebly-Avila discloses 10. (Original) The method of claim 1,
wherein the performing the comparison comprises determining that the sequence number does not correspond to the one or more-pixel values (Jeong, [0117] “On the other hand, when the drive pattern of the at least one pixel does not correspond to the preset drive pattern, the determining unit 522 may determine that a screen freeze error has occurred.”).
Regarding claim 11 Change-Jeong-Schnebly-Avila
The limitation of Claim 11 are substantially the same as those in claim 1. Therefore, the supporting rationale of the rejection to claim 1 applies equally as well to claim 11. Furthermore, regarding the claim limitation of “one or more processing units.” (Chang, [0011] “a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations.”
Regarding claim 12 Change-Jeong-Schnebly-Avila
Chang-Jeong-Schnebly-Avila disclose 12. (Original) The processor of claim 11,
wherein the one or more operations include at least one of preventing the output frame from being presented on a display device (Jeong, see FIG. 5, ¶ [0121-0122] Jeong discloses, “The output unit 526 outputs a warning in accordance with the occurrence of a screen freeze error. When it is determined that a screen freeze error has occurred, the output unit 526 may output a warning to the occupant visually, audibly, or tactually. For example, the output unit 526 may output a warning screen on the display panel. Meanwhile, the detection device 52 may determine whether a blackout occurs on the screen of the display device. Here, the blackout of the screen indicates that all pixels in the active area of the display panel are turned off or that the pixels display a black RGB value.”),
blanking a screen on the display device, or generating a warning associated with the output frame (Jeong, see FIG. 5, ¶ [0121-0122] Jeong discloses, “The output unit 526 outputs a warning in accordance with the occurrence of a screen freeze error. When it is determined that a screen freeze error has occurred, the output unit 526 may output a warning to the occupant visually, audibly, or tactually. For example, the output unit 526 may output a warning screen on the display panel. Meanwhile, the detection device 52 may determine whether a blackout occurs on the screen of the display device. Here, the blackout of the screen indicates that all pixels in the active area of the display panel are turned off or that the pixels display a black RGB value.”).
Regarding claim 17 Change-Jeong-Schnebly-Avila
Chang-Jeong-Schnebly-Avila discloses 17. (Original) The processor of claim 11,
wherein the output frame comprises at least one of a three-dimensional (3D) view of an environment captured by the one or more cameras, a two-dimensional (2D) view of the environment, a top-down view of the environment, or a plurality of viewport views associated with the environment. (Chang, [0015] “…the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from the different image sensors, and in turn, produce corresponding output frames of image data (e.g., preview display frames, still-image captures, frames for video, frames for object tracking, etc.” [0018] “The apparatus may include one, two, or more image sensors, such as including a first image sensor. When multiple image sensors are present, the first image sensor may have a larger field of view (FOV) than the second image sensor or the first image sensor may have different sensitivity or different dynamic range” [0034] “The present disclosure provides … (MF-DOF) for deblurring background regions of interest …using of two image frames obtained at two different focus points corresponding to the multiple ROIs…”).
Regarding claim 19 Chang-Jeong-Schnebly-Avila
Chang discloses Claim 19. (Currently Amended) A system comprising:
One or more processing units to determine an expected sequence number corresponding to an output frame. (Chang, [0015] “…, the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from the different image sensors, and in turn, produce corresponding output frames” additionally [0014] refers to the image signal processors (ISP) receiving a sequence of image frames and producing a flow of output frames for further processing. See also paragraph [0016]), generated from a plurality of input frames that depict one or more views captured using one or more cameras, (Chang, [0015] “After an output frame representing the scene is determined by the image signal processor using the image correction, …the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from the different image sensors, and in turn, produce corresponding output frames of image data” [0049] “…image sensors 101 and 102 to generate corrected image frames based on an image frame captured by the depth sensor 140.” i.e., it discloses input frame depth and image sensors processed similarly, which are derived from different optical path thus constituting different ‘views’ captured using multiple cameras.)
Chang does not explicitly disclose
determine whether to cause presentation of the output frame on a display device based on a comparison of the expected sequence number with a pixel value stored in a [[a]] an alpha pixel channel associated with a pixel included in the output frame, wherein a position of at least one of the one or more pixels corresponds to a position within a display device that is visible to a user;
However, in the same field of endeavor Jeong discloses more explicitly the following:
determine whether to cause presentation of the output frame on a display device based on a comparison of the expected sequence number with a pixel value . . ., . . .. (Jeong, [0115] “The determining unit 522 determines the occurrence of a screen freeze error by comparing the drive pattern of at least one pixel with a preset drive pattern.” Additionally, paragraph, [0116]-[0117] reinforce that the system checks whether the actual drive pattern matches the preset pattern for each frame. If the comparison fails, the system may suppress or block the display of the output frame, thereby preventing its presentation on the display device.”
Therefore, it would have been obvious to a person having ordinary skill in the art before
the effective filing date of the application to modify the teachings of Chang with Jeong to create the system of Chang as outlined above in order to “determine whether to cause presentation of the output frame on a display device based on a comparison of the expected sequence number with a value stored in a pixel channel associated with a pixel included in the output frame” as suggested by Jeong
The reasoning is that to “ improve detection accuracy of a screen freeze error and reliability of information transmitted to an occupant in the vehicle.” (Jeong, [0188])
Chang-Jeong does not explicitly disclose
wherein a position of at least one of the one or more pixels corresponds to a position within a display device that is visible to a user.
. . . stored in [[a ]] an alpha pixel channel associated with a pixel included in the output frame.
However, in the same field of endeavor Schnebly discloses more explicitly the following:
wherein a position of at least one of the one or more pixels corresponds to a position within a display device that is visible to a user. (Schnebly, [0006] “The code can count, or otherwise change, from one frame to the next. Verification at the destination, or display, of the changing code within the frames of the video stream confirms that the video stream is not in a fault condition.” [0008] “The sequence of video frames with the embedded code values is transmitted to a destination where the sequence of code values is extracted. A second sequence of code values is generated at the destination and then compared with the extracted code values. A fault indication is presented when the comparison does not match and the video frames are displayed when the comparison does match.”, See also Abstract)
Therefore, it would have been obvious to a person having ordinary skill in the art before
the effective filing date of the application to modify the teachings of Chang-Jeong with Schnebly to create the system of Chang-Jeong as outlined above, “wherein a position of at least one of the one or more pixels corresponds to a position within a display device that is visible to a user;” as suggested by Schnebly. In particularly, one of ordinary skill in the art would have been motivated to combine the teaching of Chang-Jeong with Schnebly in order to archive the benefit that “Extracting and verifying a sequential code can be a much more efficient operation than calculating a checksum or CRC over each frame of a video stream” (Schnebly, [0006])
Chang-Jeong-Schnebly does not explicitly disclose
. . . stored in [[a ]] an alpha pixel channel associated with a pixel included in the output frame.
However, in the same field of endeavor Avila discloses more explicitly the following:
. . . stored in [[a ]] an alpha pixel channel associated with a pixel included in the output frame. (Avila, [0178] “In this way, some implementations use the alpha channel (in texture) to store the number of accumulated samples per pixel. Each pixel has a different number of accumulated (alpha) values over time. This information is used to improve..”)
Claim Rejections - 35 USC § 103.
Claim(s) 4-5,7, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Chang-Jeong-Schnebly-Avila in view of Feng et al (US 20230012219 A1) hereinafter “Feng”.
Regarding claim 4, Chang-Jeong-Schnebly-Avila-Feng
Chang-Jeong-Schnebly-Avila discloses 4. (original) The method of claim 1, further comprising:
Chang-Jeong-Schnebly-Avila does not explicitly disclose
determining the sequence number to be associated with a set of input frames; inserting the sequence number into a set of pixels within the set of input frames; and
combining the set of input frames to generate the output frame.
However, in the same field of endeavor Feng discloses more explicitly the following:
determining the sequence number to be associated with a set of input frames; (Feng, see FIG. 1, ¶ [0015] Feng discloses, “For example, the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from the different image sensors, and in turn, produce corresponding output frames of image data (e.g., preview display frames, still-image captures, frames for video, frames for object tracking, etc.). In other examples, the image signal processor may output frames of the image data to various output devices and/or camera modules for further processing, such as for 3A parameter synchronization (e.g., automatic focus (AF), automatic white balance (AWB), and automatic exposure control (AEC)), producing a video file via the output frames, configuring frames for display, configuring frames for storage, transmitting the frames through a network connection, etc.”);
inserting the sequence number into a set of pixels within the set of input frames (Feng, see FIG. 1, [0015] “The video sequence may include other image frames received from the image sensor or other images sensors and/or other corrected image frames based on input from the image sensor or another image sensor. In some embodiments, the processing of one or more image frames may be performed within the image sensor, such as in a binning module. The image processing techniques described in embodiments disclosed herein may be performed by circuitry, such as a binning module, in the image sensor, in the image signal processor (ISP), in the application processor (AP), or a combination or two or all of these components.” Within this paragraph, Feng states the inputs of the sequence and how it’s used to produce a flow of output frames.); and
combining the set of input frames to generate the output frame (Feng, see FIG. 1, ¶ [0016] Feng discloses, “In some aspects, the corrected image frame may be produced by combining aspects of the image correction of this disclosure with other computational photography techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR)…”.).
Therefore, it would have been obvious to a person having ordinary skill in the art before
the effective filing date of the application to modify the teachings of Chang-Jeong-Schnebly-Avila with Feng to create the system of Chang-Jeong-Schnebly-Avila as outlined above in order to “determining the sequence number to be associated with a set of input frames; inserting the sequence number into a set of pixels within the set of input frames; and \combining the set of input frames to generate the output frame.” as suggested by Feng.
The reasoning is that “to improve the quality of photographs and videos captured by image capture devices.” (Feng, [0004]).
Note: The motivation that was utilized in the rejection of claim 4, applies equally as well to claims 5,7, and 15.
Regarding claim 5 Chang-Jeong-Schnebly-Avila-Feng
Chang-Jeong-Schnebly-Avila discloses 5. The method of claim 1,
further comprising combining a set of input frames to generate the output frame (see Feng, see FIG. 1, ¶ [0016] Feng discloses, “In some aspects, the corrected image frame may be produced by combining aspects of the image correction of this disclosure with other computational photography techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR).” Feng, immediately after the last paragraph, states combining aspects of image correction, which includes the set of input frames to generate an output frame.)
based at least on user input associated with a visualization of an environment captured using the one or more cameras ( Feng, see FIG. 3, ¶ [0047] Feng discloses, “The device may include or be coupled to a sensor hub 150 for interfacing with sensors to receive data regarding movement of the device 100, data regarding an environment around the device 100, and/or other non-camera sensor data…The data from sensor hub 150 may be used by the image signal processor 112 for generating corrected image frames, such as by applying electronic image stabilization (EIS) and/or digital image stabilization (DIS). [0048] “The image signal processor 112 may receive image data from one or more cameras in the form of image frames. In one embodiment, a local bus connection couples the image signal processor 112 to image sensors 101 and 102 of a first and second camera, respectively.”).
Regarding claim 7 Chang-Jeong-Schnebly-Avila-Feng
Chang-Jeong-Schnebly-Avila discloses 7. (Original) The method of claim 1,
further comprising combining a set of input frames to generate the output frame; (Feng, see FIG. 1, ¶ [0016] Feng discloses, “In some aspects, the corrected image frame may be produced by combining aspects of the image correction of this disclosure with other computational photography techniques such as high dynamic range (HDR) photography or multi-frame noise reduction (MFNR).” Feng, immediately after the last paragraph, states combining aspects of image correction, which includes the set of input frames to generate an output frame.)
based at least on at least one of a projection operation, a rotation operation, a cropping operation, a translation operation, a stitching operation, or blending operation ( Feng, see FIG. 6, ¶ [0072] Feng discloses, “The original captured first image, with its blurred background, is processed (e.g., cropped, tone mapped, etc.) by the image signal processor to determine a corrected first image frame that is output and shown on a preview or saved as a photography.” Feng discloses the operation of cropping.).
Regarding claim 15 Chang-Jeong-Schnebly-Avila-Feng
Chang-Jeong-Schnebly-Avila discloses 15. (Original) The processor of claim 11,
wherein the one or more processing units further perform operations comprising: determining the sequence number is to be associated with a set of input frames; (Feng, see FIG. 1, ¶ [0015] Feng discloses, “For example, the image signal processor may be configured to obtain input frames of image data (e.g., pixel values) from the different image sensors, and in turn, produce corresponding output frames of image data (e.g., preview display frames, still-image captures, frames for video, frames for object tracking, etc.). In other examples, the image signal processor may output frames of the image data to various output devices and/or camera modules for further processing, such as for 3A parameter synchronization (e.g., automatic focus (AF), automatic white balance (AWB), and automatic exposure control (AEC)), producing a video file via the output frames, configuring frames for display, configuring frames for storage, transmitting the frames through a network connection, etc.”);
inserting the sequence number into a set of pixels within the set of input frames; (Jeong, “Specifically, the comparing unit 524 may compare at least one of RGB information or cyclic redundancy check (CRC) information of the pixels in the porch area with a preset value. The preset value may be at least one of a preset RGB value or a preset CRC value., Feng states the inputs of the sequence and how it’s used to produce a flow of output frames.”); and
combining the set of input frames into the output frame based at least on user input associated with a visualization associated with the output frame ( Chang, [0054] “The I/O components 116 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from the user and to provide output to the user through the display 114”, see [0014]-[0015] discuss producing a flow of output frames from input frames using an image signal processor and optionally displaying ).
Claim Rejections - 35 USC § 103
Claim(s) 6 is rejected under 35 U.S.C. 103 as being unpatentable over Chang-Jeong- Schnebly-Feng in view of Hur et al. (U.S. 2020/0107008 A1) hereafter “Hur”.
Regarding claim 6 Chang-Jeong-Schnebly-Avila-Feng-Hur
Chang-Jeong- Schnebly-Feng discloses 6. (Original) The method claim 5,
Chang-Jeong- Schnebly-Feng does not explicitly disclose
wherein the visualization comprises at least one of a three-dimensional (3D) view of the environment, a two-dimensional (2D) view of the environment, a top-down view of the environment, or a contiguous composite of a plurality of viewport views associated with the environment.
However, in the same field of endeavor Hur discloses more explicitly the following:
wherein the visualization comprises at least one of a three-dimensional (3D) view of the environment, a two-dimensional (2D) view of the environment, a top-down view of the environment, or a contiguous composite of a plurality of viewport views associated with the environment. (Hur, [0094] “The rendering process may be a process of rendering and displaying the image/video data re-projected in the 3D space. Depending on the context, a combination of re-projection and rendering may be expressed as rendering on the 3D model. The image/video re-projected on the 3D model (or rendered on the 3D model) may have the form as indicated by 130 in FIG. 1. The image/video indicated by 130 in FIG. 1 is re-projected on a spherical 3D model. The user may view a portion of the rendered image/video through a VR display. At this time, the portion of the image/video viewed by the user may have the form shown in (140) of FIG. 1.” see that Paragraph-[0213]-[0215] “…tiles including the viewport area “).
Therefore, it would have been obvious to a person having ordinary skill in the art before
the effective filing date of the application to modify the teachings of Chang-Jeong-Feng with Hur to create the system of Chang-Jeong-Feng as outlined above in order to “wherein the visualization comprises at least one of a three-dimensional (3D) view of the environment, a two-dimensional (2D) view of the environment, a top-down view of the environment, or a contiguous composite of a plurality of viewport views associated with the environment” as suggested by Hur
The reasoning is that “to efficiently provide an overlay to a 360 video and efficiently display additional information based on the user's vision.” (Hur, [0632]).
Claim Rejections - 35 USC § 103
Claim(s) 3, 8-9, 13, 16, 18, 20 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Chang-Jeong-Schnebly-Avila in view of Hur et al. (U.S. 2020/0107008 A1) hereafter “Hur”.
Regarding claim 3 Chang-Jeong-Schnebly-Avila-Hur
Chang-Jeong-Schnebly-Avila discloses 3. (Original) The method of claim 2, wherein the causing the presentation comprises:
. . .
transmitting the composite frame for presentation on the display device. (Chang, [0015]“…the output frame may be displayed on a device display as a single still image and/or as part of a video sequence, saved to a storage device as a picture or a video sequence, transmitted over a network, and/or printed to an output medium.”).
Chang-Jeong-Schnebly-Avila does not explicitly discloses
generating a composite frame, at least, by blending the second output frame with an overlay;
However, in the same field of endeavor Hur discloses more explicitly the following:
generating a composite frame, at least, by blending the second output frame with an overlay; (Hur, [0488] “…alpha blending can be supported for overlay media including the alpha plane.” [0089] “…a value corresponding to A in the RGBA value may indicate an alpha channel value…the overlay and the alpha plane …be linked”)
Therefore, it would have been obvious to a person having ordinary skill in the art before
the effective filing date of the application to modify the teachings of Chang-Jeong-Schnebly-Avila with Hur to create the system of Chang-Jeong-Schnebly-Avila as outlined above in order to “generating a composite frame, at least, by blending the second output frame with an overlay” as suggested by Hur.
The reasoning is that “to improve the quality of photographs and videos captured by image capture devices”, (Chang, [0004])
Note: The motivation that was utilized in the rejection of claim 3, applies equally as well to claims 8-9, 13, 16, 18, 20 and 21.
Regarding claim 8 Chang-Jeong-Schnebly-Avila-Hur
Chang-Jeong-Schnebly-Avila discloses 8. (Original) The method of claim 1,
wherein the performing the comparison comprises comparing the sequence number (Hur, [0117] “The mfhd box (movie fragment header box) can include information about correlation between divided fragments. The mfhd box can indicate the order of divided media data of the corresponding fragment by including a sequence number. Further, it is possible to check whether there is missed data among divided data using the mfhd box.”)
with a set of alpha channel values from a set of pixel locations in the output frame, wherein the set of alpha channel values include the one or more pixel values (Hur, [0488] “The overlay media can include an alpha plane, and alpha blending can be supported for overlay media including the alpha plane. However, when the overlay media does not include the alpha plane, alpha blending may be applied on the assumption that the alpha value or the alpha channel value for all pixels of the overlay is 255, which is the maximum value, or 1.0. The overlay alpha plane for overlay media may have the same width and height resolution as the overlay media.”).
Regarding claim 9 Chang-Jeong-Schnebly-Avila-Hur
Chang-Jeong-Schnebly-Avila-Hur discloses 9. (Original)The method of claim 8,
wherein the performing the comparison further comprises retrieving the set of alpha channel values from the set of pixel locations in proximity to one or more corners of the output frame (Hur, [0549] “…the alpha plane video track may represent a video track including information about the alpha plane of the overlay…the overlay video track may represent a video track including information.” [0553] “…the overlay related metadata may include location information about a specific region…” [0549]-[0554]).
Regarding claim 13 Chang-Jeong-Schnebly-Avila-Hur
Chang-Jeong-Schnebly-Avila disclose 13. (Original) the processor of claim 11, wherein the one or more processing units further perform operations comprising:
determining a second sequence number for a second output frame (Jeong, [0133] “Based on the comparison result of the comparing unit 524, the monitoring unit 520 may monitor a drive pattern of at least one pixel in the porch area.” [0136] “The first image frame 60 includes a first back porch area 600, a first active area 602, a first front porch area 604-, and first-pixel data 601. The second image frame 62 includes a second back porch area 620, a second active area 622, a second front porch area 624-, and second-pixel data 621. The third image frame 64 includes a third back porch area 640, a third active area 642, a third front porch area 644-, and third-pixel data 641.”) that temporally succeeds the output frame; (Hur, [0140] “A DASH data model may have a hierarchical structure (410). Media presentation may be described by the MPD. The MPD may describe the temporal sequence of a plurality of periods making media presentation. One period may indicate one section of the media content. [0141] In one period, data may be included in adaptation sets. An adaptation set may be a set of media content components that can be exchanged with each other. Adaptation may include a set of representations. One representation may correspond to a media content component. In one representation, content may be temporally divided into a plurality of segments. This may be for appropriate access and delivery. A URL of each segment may be provided in order to access each segment.” see that FIG. 4).
determining, based at least on a second comparison of the second sequence number with one or more additional pixel values from the second output frame, that the second sequence number corresponds to the one or more additional pixel values; (Jeong, [0133] “Based on the comparison result of the comparing unit 524, the monitoring unit 520 may monitor a drive pattern of at least one pixel in the porch area. [0136] The first image frame 60 includes a first back porch area 600, a first active area 602, a first front porch area 604-, and first-pixel data 601. The second image frame 62 includes a second back porch area 620, a second active area 622, a second front porch area 624-, and second-pixel data 621. The third image frame 64 includes a third back porch area 640, a third active area 642, a third front porch area 644-, and third-pixel data 641.” see FIG. 5); and
based on the second sequence number corresponding to the one or more additional pixel values, causing a presentation of the second output frame on a display device (Jeong, [0137] “The display panel may sequentially output the first image frame 70, the second image frame 72, the third image frame 74, and the fourth image frame 76.” [0138] “The first pixel data 601, the second pixel data 621, and the third pixel data 641 are pixel data related to one specific pixel. The specific pixel may have a drive pattern according to the first pixel data 601, the second pixel data 621, and the third pixel data 641.”, see FIG. 8).
Regarding claim 16 Chang-Jeong-Schnebly-Avila-Hur
Chang-Jeong-Schnebly-Avila discloses 16. The processor of claim 11,
wherein the comparison comprises:
retrieving the one or more-pixel values as a set of alpha channel values from a set of pixel locations in one or more viewports within the output frame; (Hur, [0256] “The rendering process may refer to a process of rendering image/video data re-projected in a 3D space and display the rendered data. In this course, an image/voice signal may be reconfigured into a form suitable to be output. It is possible to track a viewing orientation in which a region of interest for a user is present, a viewing position/head position of the ROI, and a view point of the ROI, and necessary image/voice/text information only may be selectively used based on the information. In this case, an image signal may be selected with a different viewing position according to a user's ROI, and, as a result, an image of a specific direction from a specific viewing position at a specific viewpoint may be output.” [0489] (“…the alpha plane may refer to an alpha channel or a set of alpha channel values, and a value corresponding to A in the RGBA value may indicate an alpha channel value. The overlay alpha plane may exist in an image item or video track and may be included in overlay related metadata.”); and
comparing the sequence number with the set of alpha channel values (Hur, [0488] “The overlay media can include an alpha plane, and alpha blending can be supported for overlay media including the alpha plane. However, when the overlay media does not include the alpha plane, alpha blending may be applied on the assumption that the alpha value or the alpha channel value for all pixels of the overlay is 255, which is the maximum value, or 1.0. The overlay alpha plane for overlay media may have the same width and height resolution as the overlay media.”, also [0490] discloses how alpha plane samples are synchronized with video samples, sharing the same composition time, which may act as a proxy for timing/sequence number).
Regarding claim 18 Chang-Jeong-Schnebly-Avila-Hur
Chang-Jeong-Schnebly-Avila disclose 18. (Original) The processor of claim 11, wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources ( Hur, [0256] “In one embodiment, a region-wise packing method may be applied for overlay media packing. In the region-wise packing method, the entire region of the projected picture, in which the projection is applied to the VR media (or 360 media), can be divided into sections, and the sections can be packed in different resolutions according to importance.” see FIGS. 35B and 36,).
Regarding claim 20 Chang-Jeong-Schnebly-Avila-Hur
Chang-Jeong-Schnebly-Avila disclose 20. (Original) The system of claim 19,
wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (Hur, [0256] “…the projection is applied to the VR media (or 360 media), can be divided into sections, and the sections can be packed in different resolutions according to importance.”).
Regarding claim 21 Chang-Jeong-Schnebly-Avila-Hur
Chang-Jeong-Schnebly-Avila discloses 21. (New) The method of claim 1,
wherein, for each of the one or more pixels, the pixel channel stores information exclusive of color information associated with the pixel. (Hur, [0423] …, as can mean the alpha value of the source pixel and ad can mean the alpha value of the destination pixel. ‘s’ can mean the color (RGBA) value of the source pixel, and d can mean the color (RGBA) value of the target pixel.” Furthermore, Hur, [0487] discloses that the alph channel—a value exclusive of color information (Such as RGB)— is stored and used per pixel to determine whether or not the RGB value should be rendered).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ASTEWAYE GETTU ZEWEDE whose telephone number is (703)756-1441. The examiner can normally be reached Mo-Fr 8:30 am to 5:30 pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Vaughn can be reached at (571)272-3922. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ASTEWAYE GETTU ZEWEDE/Examiner, Art Unit 2481 /WILLIAM C VAUGHN JR/Supervisory Patent Examiner, Art Unit 2481