Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-11 is/are rejected under 35 U.S.C. 102(1) as being anticipated by Newman (US 9025896 B2).
Regarding Claim 1, Newman teaches a method, comprising: retrieving, from a storage device in a first pass, first portions of image data representative of an image (col. 4, lines 16-20: “As acquired, the Bayer sensor only collects one of the three color primaries at every pixel site--the two other primaries are predicted via a range of different algorithms that typically take substantial compute time for high quality results.”); generating, based on the first portions and without at least second portions of the image data, a first preview of the image (col. 3, lines 52-60: “A single high definition Bayer frame of 1920x1080 interleaved red, green, and blue pixels can be separated into four planes of quarter-resolution images, each consisting 960.times.540 pixels of either the red component, blue component, or one of the two green components. If red is the upper left pixel of the frame, a correlated red plane is fetched by reading every second pixel on every other scan-line. The same technique can be applied for all colors so that each plane contains the signal for one color primary.”; col. 4, lines 16-20, as above); presenting the first preview (col. 4, lines 54-56, as above); retrieving, from the storage device in a second pass, the second portions of the image data (col. 3, lines 52-60, as above); generating, based on the first portions and the second portions of the image data, a second preview of the image (col. 5, lines 58-65: “By way of a new example, a full-resolution decode mode may perform the method outlined in the following paragraphs. During the full-resolution decode mode, all four quarter-resolution planes are decoded. Any color-plane differencing is reversed so that planes of red, green1, green2 and blue are restored. The resulting planes are interleaved back into the original Bayer layout, and the result of the decode now matches the original source image.”); and presenting the second preview (col. 5, lines 65-67: “A de-Bayer operation is performed to convert the image into a full raster RGB frame and this result is presented to the calling application.”).
Regarding Claim 2, Newman teaches the method of claim 1, wherein the first portions are uniformly distributed across the image (col. 3, lines 56-58: “If red is the upper left pixel of the frame, a correlated red plane is fetched by reading every second pixel on every other scan-line.”); and the second portions are uniformly distributed across the image (col. 3, lines 58-60: “The same technique can be applied for all colors so that each plane contains the signal for one color primary.”).
Regarding Claim 3, Newman teaches The method of claim 2, wherein the image has a plurality of sections; the first portions are from the sections respectively; and the second portions are from the sections respectively (col. 3, lines 56-60, as above.).
Regarding Claim 4, Newman teaches the method of claim 3, wherein each respective section in the sections is represented by a respective portion in the first portions in the first preview (col. 3, lines 52-56, as above.).
Regarding Claim 5, Newman teaches the method of claim 4, further comprising: scaling the respective portion as an approximation of the respective section in the first preview (col. 4, lines 54-58: “For fast preview/playback the decoder will reconstruct the image at quarter resolution of the source (in this example 960.times.540), and to do this it only needs to decode Channel G, R-G and B-G to provide a standard RGB image to the requesting tool.”).
Regarding Claim 6, Newman teaches the method of claim 4, further comprising: replicating the first portions as an approximation of the second portions in the first preview (col. 5, lines 9-11: “ If the unmodified red, green1, green2, and blue planes were encoded, only one of the two green channels needs to be presented for preview.”); and replacing the approximation of the second portions in the first preview with the second portions in the generating of the second preview from the first preview and the second portions (col. 5, lines 13-17: “When color differencing is applied, the RGB planes would be reconstructed as follows: Red plane=(R-G+G) divide 2 Green plane=G divide 2 Blue plane=(B-G+G) divide 2”).
Regarding Claim 7, Newman teaches the method of claim 4, wherein the retrieving of the second portions is performed in parallel with the generating of the first preview and the presenting of the first preview (col. 5, lines 35-44: “For higher quality full-resolution presentation, the decoder performs de-Bayer filtering so the post-production tools can manipulate a traditional full-resolution image. DeBayer filtering is slow because it is highly compute intensive, and certain embodiments of the invention allow transfer of the processing from the camera to the post-production stage at which point the processing is typically performed on powerful computer workstations and is more suited to high-quality de-Bayer processing.”); and the second preview is presented to replace the first preview (col. 6, lines 24-26: “Switching between very fast preview-decode and full-resolution de-Bayer output is made automatically in one embodiment.”).
Regarding Claim 8, Newman teaches the method of claim 4, wherein each respective section in the sections is represented by a scaled version of a combination of a respective portion in the first portions and a respective portion in the second portions (col. 3, lines 52-56, as above.).
Regarding Claim 9, Newman teaches the method of claim 4, further comprising: writing, into the storage device, the first portions sequentially (col. 4, lines 6-8: “These modified image planes are encoded (e.g., compressed) just as they would if they were separate planes of R, G and B, or Y, U and V components.”); and writing, into the storage device, the second portions sequentially after the first portions (col. 6, lines 24-26, as above).
Regarding Claim 10, Newman teaches the method of claim 4, further comprising: applying an image processing operation to the first preview to generate a first result (col. 6, lines 34-41: “A lower quality, but more efficient, de-Bayer filter can be used for real-time preview during editing and a higher quality algorithm, which may be computationally slower, can be used for export (e.g., to film or a digital presentation format). Workflow is improved further because preprocessed sensor data is better for adjusting color characteristics such as white balance, contrast and saturation during post-production.”); and applying the second processing operations to the second preview to generate a second result from updating the first result (col. 6, lines 30-38: “When the de-Bayer operation is not performed in the camera, the choices for post-production image enhancement are greatly improved. For example, the selection of the specific de-Bayer filter can be made after post-production when the edited material is exported to its final presentation format. A lower quality, but more efficient, de-Bayer filter can be used for real-time preview during editing and a higher quality algorithm, which may be computationally slower, can be used for export (e.g., to film or a digital presentation format).”).
Regarding Claim 11, Newman teaches The method of claim 10, wherein the image processing operation includes image compression (col. 4, lines 6-8: “These modified image planes are encoded (e.g., compressed) just as they would if they were separate planes of R, G and B, or Y, U and V components.”), image enhancement (col. 6, lines 30-34: “When the de-Bayer operation is not performed in the camera, the choices for post-production image enhancement are greatly improved. For example, the selection of the specific de-Bayer filter can be made after post-production when the edited material is exported to its final presentation format.”), or image analytics (col. 6, lines 55-59: “By using the standard codec wrapper of these common media interfaces, even RAW data can be presented to an application by developing the image to the format requirements of the calling application.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 12-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Newman (US 9025896 B2) as applied to claims 1-11 above, and further in view of Papageorgiou (US 11404106 B2).
Regarding Claim 12, Newman teaches the method of claim 11, further comprising: generating, in an image sensing pixel array of an integrated circuit device, the image data (col. 3, lines 33-37: “The interleaved color components within a Bayer sensor are typically arranged in 2.times.2 pixel squares over the entire image with red and green on the top pair, and green and blue on the bottom of each 2.times.2 pixel array.”); programming, in a storage mode, first memory cells in a memory cell array in the integrated circuit device to store the first portions (col. 3, lines 62-65: “It is possible to encode each of the planes using common compression techniques (DCT, Wavelet, etc.) such that significant data reduction is achieved without significant quality impacts.”); and programming, in the storage mode, second memory cells in the memory cell array to store the second portions (col. 3, line 65 – col. 4, line 5: “However, more compression may be obtained by differencing the channels in the following manner: G=green plane1+green plane2 R-G=2.times.red plane-G B-G=2.times.blue plane-G D=green plane1-green plane2 (D for difference between the green planes)”). It fails to teach programming, in a synapse mode, third memory cells in the memory cell array to store weight matrices; wherein the image processing operation is based on the weight matrices.
Papageorgiou teaches programming, in a synapse mode, third memory cells in the memory cell array to store weight matrices (col. 34, lines 40-44: “Similar to the crossbar network of FIG. 36, word-lines broadcast the activations across many bit lines, where the weight elements are stored in memory cells at the crossing of each word line and bit line.”); wherein the image processing operation is based on the weight matrices (col. 32, lines 44-51: “In this approach, the weights of the neural network are stationary and stored where the calculation occurs and therefore the data movement can be reduced greatly. In terms of neural network hardware implementations with digital circuits, this could be arranged as an architecture where the memory and arithmetic units are distributed in such a way that the data storage is closer to its destination processor.”).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to incorporate weight matrices into an image processing operation that utilizes information in an array. Matrices are well-known in the art as a means by which large arrays of information can be stored and processed in an efficient manner.
Regarding Claim 13, Newman teaches a device, comprising: a first integrated circuit die having an image sensing pixel array (col. 3, lines 33-37, as above); a second integrated circuit die having a memory cell array (col. 3, lines 62-65, as above.); and a third integrated circuit die having a logic circuit (col. 5, lines 40-44: “certain embodiments of the invention allow transfer of the processing from the camera to the post-production stage at which point the processing is typically performed on powerful computer workstations and is more suited to high-quality de-Bayer processing.”) configured to: program, in a first pass and in a storage mode, first memory cells in the memory cell array to store first portions of image data representative of an image captured by the image sensing pixel array, wherein the first portions are uniformly distributed across the image (col. 3, lines 33-37, as above); and program, in a second pass and in the storage mode, second memory cells in the memory cell array to store second portions of the image data, wherein the second portions are uniformly distributed across the image (col. 3, lines 62-65, as above).
Papageorgiou further teaches programming, in a synapse mode, third memory cells in the memory cell array to store weight matrices of an image processing operation configured to process the image (col. 34, lines 40-44, as above).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to incorporate weight matrices into an image processing operation that utilizes information in an array. Matrices are well-known in the art as a means by which large arrays of information can be stored and processed in an efficient manner.
Claim 14 is identical to claim 11, except that it is dependent on claim 13 rather than claim 10. As such, it is rejected on the same basis as claim 11.
Claim(s) 15-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Newman (US 9025896 B2) and Papageorgiou (US 11404106 B2) as applied to claims 1-13 above, and further in view of Poloniewicz (US 11381729 B1).
Regarding Claim 15, Newman and Papageorgiou teach the device of claim 13, but fail to teach wherein the logic circuit is configured to use the weight matrices to perform the image processing operation to generate the first portions through image compression or image enhancement and to generate the second portions.
Poloniewicz teaches wherein the logic circuit is configured to use the weight matrices to perform the image processing operation (col. 23, lines 55-62: “To identify the edges within the digital image 902, the processing component may analyze the image 902 to detect locations in which luminance values exhibit significant change. The processing component may identify the edges within the image 902 using conventional edge detection techniques. For example, the processing component may apply a kernel matrix (e.g., a matrix of weights or multiplication factors) to the digital image 902 to detect the edges.”) to generate the first portions through image compression or image enhancement (col. 30, lines 47-67: “One or more other cameras in the smartphone may be considered as secondary cameras associated with one or more image enhancement functions. For example, the smartphone may have a telephoto lens supporting ultra-zoom options. In some example embodiments, the telephoto lens may support a zoom factor that ranges between 2× to 10×. In some more advanced embodiments, the smartphone may have an ultra-wide angle lens for enhancing the field of view of the smartphone. Additionally or optionally, in some example embodiments, the smartphone may include a depth sensor to measure the depth of background objects in comparison with primary subjects in the field of view. The smartphone may be equipped with a memory storing programming instructions corresponding to the logic illustrated in the methods 700, 800, and 1000. These programming instructions may be executable by a processor of the smartphone to carry out automatic focus selection during image capture of a subject. Since the example embodiments also provide an adaptive process for focus selection, the proposed solutions apply to a wide variety of imaging scenarios and situations.”) and to generate the second portions (col. 24, lines 1-5: “In some example embodiments, the 3×3 kernel matrix may be centered on each pixel of the image 902 in turn and multiplies the pixel values of the 3×3 region around the center pixel by the corresponding weights of the kernel matrix to generate weighted pixel values.”).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to utilize Poloniewicz’s matrix-based image processing for Newman’s invention. Such matrices are well-known in the art, and would facilitate the image processing operations required of Newman’s invention.
Regarding Claim 16, Papageorgiou teaches voltage drivers (col. 14, lines 63-66: “For example, a voltage-driver may be used to provide the input activation along the wordlines and a current readout circuit may be used to read the output from the bitlines.”); current digitizers (col. 22, lines 54-56: “Alternatively, the current can be digitized directly using a current input ADC or buffered and passed to a subsequent stage.); and an integrated circuit package enclosing the voltage drivers, the current digitizers, the first integrated circuit die, the second integrated circuit die, and the third integrated circuit die; wherein the third memory cells are connected to wordlines and bitlines (col. 49, lines 50-52: “ Alternatively, the processes, methods, or algorithms can be embodied in whole or in part using suitable hardware components, such as Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), state machines, controllers or other hardware components or devices, or a combination of hardware, software and firmware components.”); wherein the logic circuit is configured to perform operations of multiplication and accumulation in the image processing operation (col. 46, lines 46-52: “the embodiment may be based on the foundational architecture shown in FIG. 40, and is realized by replacing the weight conductance G.Math.W.sub.ij with programmable memristor elements implementing those conductance values and using the baseline memristor conductance G to implement a discharge path that enables the ratiometric charge and discharge operations.”); and wherein the logic circuit is configured to: convert, using the voltage drivers connected to the wordlines and into output currents of the third memory cells summed in the bitlines, results of bitwise multiplications of bits in an input and bits stored in the third memory cells (col. 40, lines 2-5: “The embodiment of this crossbar network may utilize unit conductance values G to convert the stored SRAM memory content into a bipolar current (push and pull current components) that is integrated by a differential integrator.”); digitize, using the current digitizers connected to the bitlines, currents in the bitlines to obtain column outputs (col. 35, line 56: “The bit line voltages are digitized by ADCs.”); and generate results of an operation of multiplication and accumulation applied to the input and the weight matrices stored in the third memory cells (col. 36, lines 1-11: “It benefits from the fact that with the activation inputs to the crossbar word-lines translated to pulse-width modulated time-domain signals, the superposition of time-constants (charging times, integration times, discharge times) implemented by various crossbar network configurations can be measured at the bit lines through time measurements, where the time-to-digital conversion can be done with reference to the same time reference that was used to generate the activations”).
Regarding Claim 17, Papageorgiou teaches the device of claim 16, wherein each respective memory cell in the memory cell array is: programmable in the synapse mode to output: a predetermined amount of current in response to a predetermined read voltage when the respective memory cell has a threshold voltage programmed to represent a value of one (col. 44, lines 30-38: “The structure involving the floating gate transistor or the FeFET is treated the same as a 2 transistor (2T) cell with one transistor acting as an access switch and the other transistor as a programmable threshold voltage transistor implementing the weight of the neural network. The programmable threshold transistors can be used either as a variable resistor in triode region of operation or as a current source in subthreshold or saturation regions of operation.”); or a negligible amount of current in response to the predetermined read voltage when the threshold voltage is programmed to represent a value of zero (col. 44, lines 30-38, as above); and programmable in the storage mode to have a threshold voltage positioned in one of a plurality of voltage regions, each representative of one of a plurality of predetermined values (col. 44, lines 30-38, as above).
Claim(s) 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Newman (US 9025896 B2) as applied to claims 1-11 above, and further in view of Stern (“Three-Dimensional Image Sensing, Visualization, and Processing Using Integral Imaging”).
Regarding Claim 18, Newman teaches an apparatus, comprising: an image sensing array (col. 3, lines 33-37, as above); a storage device configured to store image data representative of the image captured via the image sensing array (col. 3, lines 52-56, as above); a display device (col. 4, lines 54-56, as above; note that preview/playback implicitly requires a display device.); and a processor configured to perform the method of claim 1 (as claim 1 rejection). It fails to teach a lens configured to project an image onto the image sensing array, the image having a plurality of sections.
Stern teaches a lens configured to project an image onto the image sensing array, the image having a plurality of sections (Fig. 2. Conventional II with planar devices. (a) Pickup. (b) Real image display. Lenslet focal length f is assumed to be smaller than the gap g . (c) Virtual image display).
Image-projecting lenses are known in the art, as demonstrated by Stern’s use of the same in three-dimensional image sensing, visualization, and processing. Ergo, it would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to modify Newman’s apparatus with such a lens, so as to project the image generated by Newman onto the sensing array.
Regarding Claim 19, Newman and Stern teach the apparatus of claim 18. Stern further teaches wherein the processor is configured to write the first portions into the storage device at first sequential addresses and write the second portions into the storage device at second sequential addresses following the first sequential addresses (p. 10, par. 3: “Digital superresolution reconstruction methods (see for instance [63]) are ideal for processing the captured sequence of the image acquired. In [18] the inverse back projection superresolution method is applied on the sequence of elemental image arrays captured with MALT to generate improved 2-D parallel perspective images of 3-D objects.”).
Sequence-based storage devices are known in the art, as demonstrated by Stern’s use of the same in three-dimensional image sensing, visualization, and processing. Ergo, it would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to modify Newman’s apparatus with such a lens, so as to project the image generated by Newman onto the sensing array.
Regarding Claim 20, Newman and Stern teach the apparatus of claim 19. Newman further teaches wherein the processor is configured to present the first preview in parallel with generation of the second preview (col. 5, lines 35-44, as above).
Response to Amendment
The amendment filed 12/11/2025 has been entered. Claims 1-20 remain pending in the application. Applicant’s amendments to the Specification, Drawings, and Claims have overcome each and every objection and 112(b) rejection previously set forth in the Non-final Office Action mailed 8/11/2025.
Response to Arguments
Applicant's arguments filed 12/11/2025 have been fully considered but they are not persuasive.
Applicant’s arguments (see applicant’s response, p. 1, “Objections to Specifications and Claims”), filed 12/11/2025, with respect to the Specification have been fully considered and are persuasive. The objection of the Specification has been withdrawn.
With respect to Newman (see applicant’s response, pp. 1-3, “Rejections under 35 U.S.C. § 102”), applicant argues that Newman does not disclose the feature of “generating, based on the first portions [of the image data of the image] and without at least second portions of the image data [of the image], a first preview of the image”. Applicant argues that the portion of Newman originally cited (col. 4, lines 54-56) fails to teach that the image reconstructed at quarter resolution that Newman teaches is constructed without “second portions” of the source image data.
The examiner has amended the relevant portion of the claim 1 rejection to cite earlier segments of Newman which clearly state that the quarter-resolution image initially cited is constructed by collecting only one of the three color primaries at every pixel site. Col. 3, lines 52-60 explicitly states that only every other pixel on every second line is read. This shows collection of first portions of data without at least a second portion. The examiner considers these color primaries a “first portion of the image data”; given that Newman explicitly excludes the other color primaries in this initial image preview, this is considered to be “without at least second portions of the image data”. Accordingly, Newman anticipates the inventions recited in claim 1 and its dependent claims.
Applicant’s other arguments (see applicant’s response, p. 4, “Rejections under 35 U.S.C. § 103”) depend on applicant’s arguments regarding the rejection of claim 1 in light of Newman. As such, these arguments are also not persuasive.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN A BARHAM whose telephone number is (571)272-4338. The examiner can normally be reached Mon-Fri, 8:30am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu, can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN ALLEN BARHAM/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights