DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Applicant’s Amendments filed on 07/08/2025 has been entered and made of record.
Currently pending Claim(s)
1, 3–4, 6–7, 9, 12–18
Independent Claim(s)
1, 7, 14
Amended Claim(s)
1, 3, 7, 9, 12, 17
Canceled Claim(s)
2, 5, 8, 10–11
Response to Arguments
This office action is responsive to Applicant’s Arguments/Remarks Made in an Amendment received on July 8, 2025.
In view of the new claim amendments and applicant arguments, Remarks, filed on July 8, 2025 with respect to 35 U.S.C. 112(b) claim rejection have been carefully considered and the claim rejection under 35 U.S.C. 112(b) is withdrawn.
On bottom of page, 8, applicant argues, in summary, the applied prior art of Song et al. ('CN 110070500 B’) does not teach or suggest the following limitations described in independent claim 1:
“identifying a plurality of second valid points located around a first point among the first invalid points from the disparity image data.”
However, the Examiner respectfully disagrees with Applicant’s line of reasoning. The Examiner has thoroughly reviewed the Applicant’s arguments but respectfully believes the cited reference to reasonably and properly meet the claimed limitations. Song discloses identifying second valid points around first invalid points, which are used to fill first invalid points in the disparity image data to generate first image data, [¶0026]. Song initially starts with a disparity map Disp and uses the disparity value of the corresponding position of the copy DispCopy to fill in points and takes into account the adjacent pixels by using a filling template [¶0020 & 0022]. With a broad interpretation, Song discloses identifies second valid points that are later used to fill in first invalid points, [¶0024 & 0026].
Further, on page 8, applicant argues, in summary, the applied prior art of Ooi et al. (‘US 2013/0021332 A1’) does not teach or suggest the following limitations described in independent claims 1 and 14:
“filling corresponding one among the plurality of second invalid points according to a plurality of first valid points in a direction of the corresponding blocked region of the first image data.”
Regarding the “filling” limitation with a broad interpretation, the Examiner understands Ooi teaches filing filling invalid points based on valid points in a direction of the corresponding blocked region. With a broad interpretation, the left or right direction can be considered a direction of the corresponding blocked region. Once the valid points are identified, the invalid points are substituted or filled with the valid point values, [Figure 1 & ¶0071].
Further, starting on bottom of page 8 and continuing onto page 9, applicant argues, in summary, the applied prior art of Huang et al. (‘US 10,552,970 B2’) does not teach or suggest the following limitations described in amended independent claims 1:
“if a number of the second valid points is greater than or equal to a predetermined value, filling the first point according to the second valid points.”
As for the “predetermined value” limitation, the Examiner understands Huang to teach filling a first point according to the second valid points if the number of second valid points is greater than or equal to a predetermined value. With a broad interpretation, if a point has a threshold number or predetermined value of neighboring pixels with a valid depth value (i.e., valid point), the point’s depth value is replaced. This can be replaced based on the depth value of the neighboring pixels which teaches filing the first point according to the second valid points, [Col 4, lines 44-49].
Further, on page 9, applicant argues in summary, the applied prior art of Petrichkovich et al. (‘US 10,356,385 B2’) does not teach or suggest the following limitations described in amended independent claim 7:
“wherein label values of a plurality of pixels in an oblique direction of the invalid image data are read via a data read line in the register.”
However, the Examiner respectfully disagrees because Gao et al. (‘CN 111724431 A’) teaches reading label values via a data read line in the register, [¶0073-0074] while Petrichkovich teaches the pixels in an oblique direction, [Col 13, line 30-31]. Thus, it would be have been obvious to someone of ordinary skill in the art to modify Gao and incorporate the teachings of Petrichkovich by including oblique directions to obtain valid results with more information.
Thus, the Examiner maintains that the limitation as presented and as rejected were properly and adequately met. The rejection as presented in the non-final rejection is maintained as previously disclosed. Additional citations and/or modified citations may be present to more concisely address limitations.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 3–4 are rejected under 35 U.S.C. 103 as being unpatentable over Song et al. (CN 110070500 B) (hereafter, “Song”) in view of Alessandrini et al. (US 8,817,069 B2) (hereafter, “Alessandrini”) and further in view of Ooi et al. (US 2013/0021332 A1) (hereafter, “Ooi”) and Huang et al. (US 10,552,970 B2) (hereafter, “Huang”).
Regarding claim 1, Song discloses a disparity image filling method [this method can effectively fill holes ... a depth image post-processing method, para 005, 006], comprising: filling a plurality of first invalid points in disparity image data to generate first image data [the circular filling template C can well fill the small holes in the disparity map with the disparity values of adjacent pixels, para 0020];
filling a plurality of second invalid points [after the small holes are filled, there are still some large holes in the disparity map ... the present invention designs a 15*15 square closed template S, which can not only fill larger-sized cavities, para 0022] in the first image data to generate second image data [you need to copy the disparity map Disp first to get a copy of DispCopy, and the next operation will be performed on DispCopy, para 0022], wherein, the first invalid points are filled by searching for valid points in the disparity image data according to a first step size [aiming at the sparse holes in the disparity map Disp, a 5*5 circular filing template C is designed ... the circular filling template C can well fill the small holes in the disparity map with the disparity values of adjacent pixels, para 0020], wherein the first step size is smaller than the second step size [aiming at the sparse holes in the disparity map Disp, a 5*5 circular filling template C is designed ... a 15*15 square closed template S, which can not only fill larger-sized cavities, para 0020, 0022]; wherein, the filling of the plurality of second invalid points in the first image data [after the small holes are filled, there are still some large holes in the disparity map ... the present invention designs a 15*15 square closed template S, which can not only fill larger-sized cavities, para 0022]; wherein the filling of the plurality of first invalid points in disparity image data to generate the first image data [the circular filling template C can well fill the small holes in the disparity map with the disparity values of adjacent pixels, para 0020] comprises: identifying a plurality of second valid points located around a first point among the first invalid points from the disparity image data [for pixels with no disparity value in the disparity map Disp, use the disparity value of the corresponding position of the copy DispCopy to fill ... when Disp(x,y) is equal to 0, that is, when the pixels in the disparity map have no valid disparity, the corresponding pixel disparity value DispCopy(x,y) is assigned to Disp(x,y), para 0026].
Song fails to explicitly disclose the second invalid points are filled by searching for the valid points in the first image data according to a second step size; determining whether the plurality of second invalid points in the first image data are located in a corresponding blocked region between a first blocked region and a second blocked region; filling corresponding one among the plurality of second invalid points according to a plurality of first valid points in a direction of the corresponding blocked region of the first image data; if a number of the second valid points is greater than or equal to a predetermined value, filing the first point according to the second valid points; and if the number of the second valid points is smaller than the predetermined value, not filling the first point.
However, Alessandrini teaches the second invalid points are filled by searching for the valid points in the first image data according to a second step size [the search for a valid pixel situated after the invalid pixel Pi is concerned is effected in a fixed range defined by a plurality of pixels, for example 50 pixels ... if such a pixel P1 exists (step E7, ‘1’), there is then determined in the step E8 a vicinity N1 of pixels around the pixel P1, after which there is calculated in the step E9 the depth (or disparity) value, noted Vcal(Pi), to be assigned to the invalid pixel Pi, Col 7, line 45-47; Col 7, line 49-53].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Song and incorporate the teachings of Alessandrini to create local uniformization, as recognized by Alessandrini.
Neither Song nor Alessandrini appears to explicitly disclose determining whether the plurality of second invalid points in the first image data are located in a corresponding blocked region between a first blocked region and a second blocked region; filling corresponding one among the plurality of second invalid points according to a plurality of first valid points in a direction of the corresponding blocked region of the first image data; if a number of the second valid points is greater than or equal to a predetermined value, filing the first point according to the second valid points; and if the number of the second valid points is smaller than the predetermined value, not filling the first point.
However, Ooi teaches determining whether the plurality of second invalid points in the first image data are located in a corresponding blocked region between a first blocked region and a second blocked region [areas to the left and right of the occlusion area are searched and left and right disparity values are acquired, para 0042]; filling corresponding one among the plurality of second invalid points according to a plurality of first valid points in a direction of the corresponding blocked region of the first image data [identifies the disparity of the pixels in the occlusion area in accordance with a magnitude relation between disparity values of the left and right pixels that are adjacent to the occlusion area ... the areas to the left and right of the occlusion area are searched and disparity effective areas that are adjacent to the occlusion area are identified. Then, among the disparity values of the left and right disparity effective areas, the disparity value indicating the deeper depth is substitute for the disparity value of the occlusion area, para 0052, 0071].
Neither Song, Alessandrini, nor Ooi appears to explicitly disclose if a number of the second valid points is greater than or equal to a predetermined value, filing the first point according to the second valid points; and if the number of the second valid points is smaller than the predetermined value, not filling the first point.
Huang discloses if a number of the second valid points is greater than or equal to a predetermined value, filling the first point according to the second valid points [if the pixel is determined to have at least the threshold number of neighboring pixels with a valid depth value, the one or more processors are configured to replace the pixel's depth value with a filtered depth value. The filtered depth value can be based on the depth values of the neighboring pixels with valid depth values, Col 4, line 44-49]; and if the number of the second valid points is smaller than the predetermined value, not filling the first point [a case where the filtered depth map smoothing logic 208 of FIG. 2 would not replace the pixel’s depth value with a filtered depth value. This is because the number of valid depth values (3) is not greater than the threshold number of neighboring pixels (12 or 13), Col 11, line 19-23].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Song in view of Alessandrini and incorporate the teachings of Ooi and Huang for edge preservation and foreground-background distinction, as recognized by Ooi and Huang.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Huang and Ooi with Song and Alessandrini to obtain the invention as specified in claim 1.
Regarding claim 3, which claim 1 is incorporated, neither Song nor Alessandrini appears to explicitly disclose wherein the determining of whether the plurality of second invalid points in the first image data are located in the corresponding blocked region between the first blocked region and the second blocked region comprises: identifying a second valid point in a first direction from the corresponding one among the plurality of second invalid points toward the first blocked region; performing searching in a second direction from the corresponding one toward the second blocked region for a third valid point; and comparing the second valid point and the third valid point to determine whether the corresponding one is located in the first blocked region or the second blocked region.
Ooi teaches wherein the determining of whether the plurality of second invalid points in the first image data are located in the corresponding blocked region between the first blocked region and the second blocked region [Figure 1; areas La and Lb of disparity maps L and R shown in Fig 1 are occlusion areas, namely, areas that cannot be visually recognized by a camera because an object exists in the foreground ... in the occlusion areas, generally, there is a high possibility that disparity values are not obtained, para 0039] comprises: identifying a second valid point in a first direction from the corresponding one among the plurality of second invalid points toward the first blocked region [areas to the left (the examiner interprets the left as the first direction) and right of the occlusion area are searched and left and right disparity values are acquired. A disparity value that indicates a deeper depth is identified based on a magnitude relation between the acquired left and right disparity values, para 0042]; performing searching in a second direction from the corresponding one toward the second blocked region for a third valid point [areas to the left and right (the examiner interprets the right as the second direction) of the occlusion area are searched and left and right disparity values are acquired. A disparity value that indicates a deeper depth is identified based on a magnitude relation between the acquired left and right disparity values, para 0042]; and comparing the second valid point and the third valid point to determine whether the corresponding one is located in the first blocked region or the second blocked region [compares the magnitudes of the disparity values, Left_dspval and Right_dspval, of the left and right pixels that are adjacent to the occlusion area. When the determination result is that the value of Left_dspval is smaller than the value of Right_dspval, the generation portion 110 determines that the left side pixel that is adjacent to the occlusion area is the background and the right side pixel that is adjacent to the occlusion area is the foreground, para 0066].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Song in view of Alessandrini and incorporate the teachings of Ooi to avoid overwriting the foreground with the background, as recognized by Ooi.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Huang with Song, Alessandrini, and Ooi to obtain the invention as specified in claim 3.
Regarding claim 4, which claim 3 is incorporated, neither Song nor Alessandrini appears to explicitly disclose wherein the first direction and the second direction are opposite each other.
Ooi teaches wherein the first direction and the second direction are opposite each other [extracts disparity values based on a displacement amount in the horizontal direction of the L image and the R image (the examiner interprets the left and right to be opposite each other), para 0051].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Song in view of Alessandrini and incorporate the teachings of Ooi to reduce disparity extraction error, as recognized by Ooi.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Huang with Song, Alessandrini, and Ooi to obtain the invention as specified in claim 4.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Song (CN 110070500 B) in view of Alessandrini (US 8,817,069 B2) and further in view of Ooi (US 2013/0021332 A1) and Huang (US 10,552,970 B2), as applied above, and Nakatani et al. (US 9,684,964 B2) (hereafter, “Nakatani”).
Regarding claim 6, which claim 1 is incorporated, neither Song, Alessandrini, Ooi, nor Huang appears to explicitly disclose performing median filtering on the second image data to generate output image data.
Nakatani teaches performing median filtering on the second image data to generate output image data [the post-filter unit 110 performs the post-filter processing to eliminate anomaly values of a disparity image obtained by the merging unit 108 ... the post-filter unit 110 performs the post-filter processing by using, for example, a median filter, Col 7, line 19-21; Col 7 line 23-24].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Song in view of Alessandrini and further in view of Ooi and Huang and incorporate the teachings of Nakatani to eliminate anomaly values in the disparity image, as recognized by Nakatani.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Nakatani with Song, Alessandrini, Ooi, and Huang to obtain the invention as specified in claim 6.
Claims 7, 9, 12–13 are rejected under 35 U.S.C. 103 as being unpatentable over Alessandrini et al. (US 8,817,069 B2) (hereafter, “Alessandrini”) in view of Chen et al. (CN 114372928 A) (hereafter, “Chen”) and further in view of Gao et al. (CN 111724431 A) (hereafter, “Gao”) and Petrichkovich et al. (US 10,356,385 B2) (hereafter, “Petrichkovich”).
Regarding claim 7, Alessandrini discloses an image processing device [a device for digitally processing a depth or disparity map, Col 5, line 43-44], comprising: a first image filling circuit [here the expression computer system means any processor-based device adapted to process digital data, Col 5, line 52-54], filling a plurality of first invalid points in disparity image data to generate first image data [a vicinity N1 of pixels around the pixel P1, after which there is calculated in the step E9 the depth (or disparity) value, noted Vcal(Pi), to be assigned to the invalid pixel Pi ... a device of the invention for digitally processing a depth or disparity map includes a module 63 for filling occluded areas of the map, Col 7, line 50-53; Col 8, line 60-62]; wherein, the first invalid points are filled by searching for valid points in the disparity image data according to a first step size [the search for a valid pixel situated after the invalid pixel Pi is concerned is effected in a fixed range defined by a plurality of pixels, for example 50 pixels ... if such a pixel P1 exists (step E7, ‘1’), there is then determined in the step E8 a vicinity N1 of pixels around the pixel P1, after which there is calculated in the step E9 the depth (or disparity) value, noted Vcal(Pi), to be assigned to the invalid pixel Pi, Col 7, line 45-47; Col 7, line 49-53], wherein the first step size is smaller than the second step size [The search for a valid pixel situated after the invalid pixel Pi concerned is effected in a fixed range defined by a plurality of pixels, for example 50 pixels, that the user of the method of the invention is able to modify (the examiner interprets the user’s ability to modify as being able to make the first step size smaller than the second step size), Col 7, line 45-48].
Alessandrini fails to explicitly disclose a second image filling circuit, filling a plurality of second invalid points in the first image data to generate second image data; and the second invalid points are filled by searching for the valid points in the first image data according to a second step size; a disparity determination circuit, searching for the plurality of first invalid points in the disparity image data to generate invalid image data; and a buffer, storing the disparity image data and the invalid image data, wherein the buffer comprises: a static random access memory (SRAM), storing the disparity image data; and a register, storing the invalid image data, and wherein label values of a plurality of pixels in an oblique direction of the invalid image data are read via a data read line in the register.
However, Chen teaches a second image filling circuit [the processor 41 can implement the functions of each module ... processor 41 may be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application specific integrated circuit, para 00123, 00126], filling a plurality of second invalid points in the first image data to generate second image data [the writing module 33 is configured to recombine the valid pixel value and the second invalid pixel value to obtain second image data, para 00106]; and the second invalid points are filled by searching for the valid points in the first image data according to a second step size [calculate the number of valid pixel values in the first image data according to the width value and the N value corresponding to the copy frame, wherein the width value represent the number of pixels included in a row of image data ... if the step size is set to 5 and the copy box width is 16, then the start coordinates of other work items are (5,), (10,0), (15,0), para 00111, 0076]; a static random access memory (SRAM), storing the disparity image data [the memory 42 may include at least one type of storage medium, such as ... static random access memory (Static Random Access Memory, SRAM) ... that can be used to carry or store desired program code in the form of instructions or data structures, para 00127].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Alessandrini and incorporate the teachings of Chen to improve the image quality and store program code in the form of instructions or data structures, as recognized by Chen.
Neither Alessandrini nor Chen appears to explicitly disclose a disparity determination circuit, searching for the plurality of first invalid points in the disparity image data to generate invalid image data; and a buffer, storing the disparity image data and the invalid image data, wherein the buffer comprises: a register, storing the invalid image data, and wherein label values of a plurality of pixels in an oblique direction of the invalid image data are read via a data read line in the register.
However, Gao teaches a disparity determination circuit, searching for the plurality of first invalid points in the disparity image data to generate invalid image data [in step S140, if the similarity corresponding to the pixel P determined in step S130 is lower than a threshold, the pixel P can be regarded as an invalid pixel ... the pixel P can be regarded as an invalid pixel ... the example method may be applied at an electronic device, and may include steps S110, S120, S130, and S140, para 0038, 0023]; and a buffer, storing the disparity image data and the invalid image data [the buffer memory BUF may include any appropriate type of memory, register and/or cache memory, and may be configured to cache the first image and the second image captured by the binocular camera and the initial disparity map corresponding to the first image and the second image, para 0068]; and a register, storing the invalid image data [the image block determination unit BDU may include one or more registers for storing data or instructions, etc., para 0073]; [and wherein label values of a plurality of pixels in an oblique direction of the invalid image data] are read via a data read line in the register [the image block determination unit BDU may include one or more registers for storing data or instructions, etc. ... the image block determination unit BDU can read image data from the buffer memory BUF ... and can read disparity data from the buffer memory BUF, para 0073, 0074].
Neither Alessandrini, Chen, nor Gao appears to explicitly disclose label values of a plurality of pixels in an oblique direction of the invalid image data.
Petrichkovich teaches label values of a plurality of pixels in an oblique direction of the invalid image data [Fig 9; block cores are median filters on the nearest valid pixels from 8 directions ... anisotropic filtering, when, in order to receive a valid result, pixels shall be valid simultaneously in one of 4 pairs of opposite directions ... several lines of the left and right source images are read using the DMA controller (3) from the external memory (2) and are stored in the local memory (4), Col 13, line 30-31, line 36-38; Col 6, line 5-8].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Alessandrini in view of Chen and incorporate the teachings of Gao and Petrichkovich to obtain store intermediate and final results, as recognized by Gao and Petrichkovich.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Gao and Petrichkovich with Alessandrini and Chen to obtain the invention as specified in claim 7.
Regarding claim 9, which claim 7 is incorporated, neither Alessandrini nor Chen appear to explicitly disclose wherein the first image filling circuit updates the invalid image data stored in the buffer according to the first image data.
However, Gao teaches wherein the first image filling circuit updates the invalid image data stored in the buffer according to the first image data [the buffer memory BUF can also cache control instructions for controlling units/circuits/modules such as ... other elements/components not shown in the device to perform corresponding operations, intermedia data involved in the processing process and data to be output as processing results, para 0069].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Alessandrini in view of Chen and incorporate the teachings of Gao for continued processing, as recognized by Gao.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Petrichkovich with Alessandrini, Chen, and Gao to obtain the invention as specified in claim 9.
Regarding claim 12, which claim 7 is incorporated, neither Alessandrini nor Chen appear to explicitly disclose wherein the second image filling circuit searches for valid points in the first image data according to the invalid image data in the buffer.
However, Gao teaches wherein the second image filling circuit searches for valid points in the first image data according to the invalid image data in the buffer [the buffer memory BUF may include any appropriate type of memory, register, and/or cache memory, and may be configured to cache the first image and the second image ... and the initial disparity map corresponding to the first and second image ... the image block determination unit BDU can read image data from the buffer memory BUF ... and can read disparity data from the buffer memory BUF, para 0068, 0074].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Alessandrini in view of Chen and incorporate the teachings of Gao to read and perform processing with the buffer, as recognized by Gao.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Petrichkovich with Alessandrini, Chen, and Gao to obtain the invention as specified in claim 12.
Regarding claim 13, which claim 7 is incorporated, Alessandrini discloses a first memory, storing the disparity image data [the method of the invention for filling occluded areas of a depth/disparity map is implemented by a computer program executed in a data processing system, the map to be processed being stored beforehand in the data processing system, Col 6, line 56-60]; wherein, the first data is data is data of invalid image data arranged in a first slope direction [the direction in which a line of pixels of the map is processed corresponds to the right to left direction ... if such a pixel P0 exists (E5, ‘1’), a search is then conducted (step E6) by processing the current line in the predefined direction for a first valid pixel P1 that follows the invalid pixel Pi, Col 6, line 66-67; Col 7, line 36-38], and the invalid image data labels a plurality of invalid points in the disparity image data [each pixel of the map is associated with a depth or disparity value called the pixel value ... within this range of values, some are valid depth/disparity values and a particular value is an invalid value ... thus a pixel that corresponds to an occluded area of the map is identifiable as being invalid from the associate pixel value, Col 6, line 45-47; Col 6, line 49-50; Col 6, line 53-55].
Alessandrini fails to explicitly disclose a second memory, storing first data; and a third memory, storing second data; the second image data is data of the invalid image data arranged in a second slope direction.
However, Chen teaches a second memory, storing first data [the memory 42 in this embodiment of the present application may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and/or data, para 00127].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Alessandrini and incorporate the teachings of Chen for the processor to call the data stored in the memory, as recognized by Chen.
However, neither Alessandrini, Chen, nor Gao appears to explicitly disclose a third memory, storing second data; the second image data is data of the invalid image data arranged in a second slope direction.
Petrichkovich teaches a third memory, storing second data [the disparity map is computed via a semi-global block matching algorithm (SGBM) with post-filtering, storing intermedia and final results in the local memory, Col 6, line 11-13]; the second image data is data of the invalid image data arranged in a second slope direction [Fig 9; block cores are median filters on the nearest valid pixels from 8 directions ... anisotropic filtering, when, in order to receive a valid result, pixels shall be valid simultaneously in one of 4 pairs of opposite directions, Col 13, line 30-31, Col 13, line 36-38].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Alessandrini in view of Chen and further in view of Gao and incorporate the teachings of Petrichkovich to be able to store large volumes of data, as recognized by Petrichkovich.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Petrichkovich with Alessandrini, Chen, and Gao to obtain the invention as specified in claim 13.
Claims 14-18 are rejected under 35 U.S.C. 103 as being unpatentable over Alessandrini (US 8,817,069 B2) (hereafter, “Alessandrini”) in view of Ooi et al. (US 2013/0021332 A1) (hereafter, “Ooi”).
Regarding claim 14, Alessandrini discloses a disparity image filling method [a method of filling occluded areas of a depth or disparity map, Col 4, line 16-17].
Alessandrini fails to explicitly disclose determining whether a plurality of invalid points in first image data are located in a corresponding blocked region between a first blocked region and a second blocked region; and filling corresponding one among the plurality of invalid points according to a plurality of first valid points in a direction of the corresponding blocked region in the first image data to generate second image data.
However, Ooi teaches determining whether a plurality of invalid points in first image data are located in a corresponding blocked region between a first blocked region and a second blocked region [areas to the left and right of the occlusion area are searched and left and right disparity values are acquired, para 0042]; and filling corresponding one among the plurality of invalid points according to a plurality of first valid points in a direction of the corresponding blocked region in the first image data to generate second image data [identifies the disparity of the pixels in the occlusion area in accordance with a magnitude relation between disparity values of the left and right pixels that are adjacent to the occlusion area, para 0052].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Alessandrini and incorporate the teachings of Ooi to generate a high quality image, as recognized by Ooi.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ooi with Alessandrini to obtain the invention as specified in claim 14.
Regarding claim 15, which claim 14 is incorporated, Alessandrini fails to explicitly disclose wherein the determining of whether the plurality of invalid points in the first image data are located in the corresponding blocked region between the first blocked region and the second blocked region comprises: identifying a second valid point in a first direction from the corresponding one among the plurality of invalid points toward the first blocked region; performing searching in a second direction from the corresponding one toward the second blocked region for a third valid point; and comparing the second valid point and the third valid point to determine whether the corresponding one is located in the first blocked region or the second blocked region.
However, Ooi teaches wherein the determining of whether the plurality of invalid points in the first image data are located in the corresponding blocked region between the first blocked region and the second blocked region [Figure 1; areas La and Lb of disparity maps L and R shown in Fig 1 are occlusion areas, namely, areas that cannot be visually recognized by a camera because an object exists in the foreground ... in the occlusion areas, generally, there is a high possibility that disparity values are not obtained, para 0039] comprises: identifying a second valid point in a first direction from the corresponding one among the plurality of invalid points toward the first blocked region [areas to the left (the examiner interprets the left as the first direction) and right of the occlusion area are searched and left and right disparity values are acquired. A disparity value that indicates a deeper depth is identified based on a magnitude relation between the acquired left and right disparity values, para 0042]; performing searching in a second direction from the corresponding one toward the second blocked region for a third valid point [areas to the left and right (the examiner interprets the right as the second direction) of the occlusion area are searched and left and right disparity values are acquired. A disparity value that indicates a deeper depth is identified based on a magnitude relation between the acquired left and right disparity values, para 0042]; and comparing the second valid point and the third valid point to determine whether the corresponding one is located in the first blocked region or the second blocked region [compares the magnitudes of the disparity values, Left_dspval and Right_dspval, of the left and right pixels that are adjacent to the occlusion area. When the determination result is that the value of Left_dspval is smaller than the value of Right_dspval, the generation portion 110 determines that the left side pixel that is adjacent to the occlusion area is the background and the right side pixel that is adjacent to the occlusion area is the foreground, para 0066].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Alessandrini and incorporate the teachings of Ooi to avoid overwriting the foreground with the background, as recognized by Ooi.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ooi with Alessandrini to obtain the invention as specified in claim 15.
Regarding claim 16, which claim 15 is incorporated, Alessandrini fails to explicitly disclose wherein the first direction and the second direction are opposite to each other.
However, Ooi teaches wherein the first direction and the second direction are opposite to each other [extracts disparity values based on a displacement amount in the horizontal direction of the L image and the R image (the examiner interprets the left and right to be opposite each other), para 0051].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Alessandrini and incorporate the teachings of Ooi to reduce disparity extraction error, as recognized by Ooi.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ooi with Alessandrini to obtain the invention as specified in claim 16.
Regarding claim 17, which claim 14 is incorporated, Alessandrini fails to explicitly disclose wherein the filling corresponding one among the plurality of invalid points according to a plurality of first valid points in a direction of the corresponding blocked region in the first image data to generate second image data comprises: identifying the plurality of first valid points in a plurality of directions from the corresponding one among a plurality of second invalid points toward the corresponding blocked region, wherein the plurality of second invalid point is identified in a first direction from the corresponding one among the plurality of invalid points toward the first blocked region; and filling the corresponding one among the plurality of second invalid points according to the data values of the plurality of first valid points.
However, Ooi teaches wherein the filling corresponding one among the plurality of invalid points according to a plurality of first valid points in a direction of the corresponding blocked region in the first image data to generate second image data [identifies the disparity of the pixels in the occlusion area in accordance with a magnitude relation between disparity values of the left and right pixels that are adjacent to the occlusion area, para 0052] comprises: identifying the plurality of first valid points in a plurality of directions from the corresponding one among a plurality of second invalid points toward the corresponding blocked region [areas to the left and right (the examiner interprets the left and right to be a plurality of directions) of the occlusion area are searched and left and right disparity values are acquired. A disparity value that indicates a deeper depth is identified based on a magnitude relation between the acquired left and right disparity values, para 0042] wherein the plurality of second invalid point is identified in a first direction from the corresponding one among the plurality of invalid points toward the first blocked region [areas to the left (the examiner interprets the left as the first direction) and right of the occlusion area are searched and left and right disparity values are acquired. A disparity value that indicates a deeper depth is identified based on a magnitude relation between the acquired left and right disparity values, para 0042]; and filling the corresponding one among the plurality of second invalid points according to the data values of the plurality of first valid points [identifies the disparity of the pixels in the occlusion area in accordance with a magnitude relation between disparity values of the left and right pixels that are adjacent to the occlusion area, para 0052].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Alessandrini and incorporate the teachings of Ooi to generate a high quality image, as recognized by Ooi.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ooi with Alessandrini to obtain the invention as specified in claim 17.
Regarding claim 18, which claim 17 is incorporated, Alessandrini discloses sorting the data values of the plurality of the first valid points to select one with a median value, and setting the data value of the corresponding one to be the median value [Fig 4; pixel values of the target pixel and neighboring pixels are extracted, arranged in a row, and a pixel value (20) of the target pixel is replaced with a median (2) of nine pixel values, Col 7, line 28-31].
Alessandrini fails to explicitly disclose discloses wherein the filling corresponding one among the plurality of second invalid point according to the data values of the plurality of the first valid point.
However, Ooi teaches discloses wherein the filling corresponding one among the plurality of second invalid point according to the data values of the plurality of the first valid point [identifies the disparity of the pixels in the occlusion area in accordance with a magnitude relation between disparity values of the left and right pixels that are adjacent to the occlusion area, para 0052].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Alessandrini and incorporate the teachings of Ooi to generate a high quality image, as recognized by Ooi.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ooi with Alessandrini to obtain the invention as specified in claim 18.
Conclusion
The art made of record and not relied upon is considered pertinent to applicant's disclosure:
Five-Direction Occlusion Filling with Five Layer Parallel Two-Stage Pipeline for Stereo Matching with Sub-Pixel Disparity Map Estimation to Ma et al. discloses a multi-directional disparity occlusion filing method.
US 10,165,248 B2 to Hung et al. discloses an optimization method to repair a depth map based on a left image and a right image with a hole filling process to fill hole in the depth map.
US 2013/0010073 A1 to Do et al. discloses a system and method for generating a disparity map and associated depth map, fusing the disparity map with the captured image and filling invalid disparities.
Anisotropic Median Filtering for Stereo Disparity Map Refinement to Einecke et al. discloses refining stereo disparity maps with median filtering and edge-preserving anisotropic filtering.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TOLUWANI MARY-JANE IJASEUN whose telephone number is (571)270-1877. The examiner can normally be reached Monday - Friday 7:30AM-4PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached on (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TOLUWANI MARY-JANE IJASEUN/Examiner, Art Unit 2676