Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on November 6th, 2025 is being considered by the examiner.
Response to Amendment
The Amendment filed 12/31/2025 has been entered. Claims 1, 3, 5-14, 16, 18, 19, 21, and 23, 24, and 26 remain pending. Claims 2, 4, 15, 17, 20, 22, and 25 are cancelled.
In response to the amended abstract filed, the objection of record with respect to the Specification is withdrawn.
In response to the amendment to Claim 3, the objection of record with respect to the Claims is withdrawn.
In response to the amendment to Claims 1 and 3, the rejection under 35 U.S.C. 112(b) is withdrawn.
Response to Arguments
Applicant’s arguments with respect to claims 1, 3, 6-10, 18, 19, 23, and 24 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Objections
Claim 13 is objected to because of the following informalities: “wherein the each sub-image sensor” (emphasis added) should be amended to “wherein each sub image sensor.” Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3, 5-10, 16, 18, 19, 21, 23, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Pau (US 2021/0183085 A1), in view of Mor (US 2019/0068951 A1), further in view of Yang et al. (CN 103780844 A).
Regarding Claim 1, Pau discloses “A depth data computing device” (Pau, [0016] discloses “The disclosed embodiments relate to methods, devices and systems that apply digital fringe projection (DFP) techniques to facilitate the generation of three-dimensional (3D) images of an object based on the measurement of polarizations and/or colored light in a single shot”), “comprising:
a projection device for scanning and projecting a set of structured light having different patterns to a shooting area, and the set of structured lights includes at least two structured lights of different patterns” (Pau, [0017] discloses “DFP methods typically utilize a projector, such as a digital light processing (DLP), liquid crystal display (LCD) or a liquid crystal on silicon (LCoS) projector, to project computer generated fringe patterns onto an object”; where computer generated fringe patterns are structured light having different patterns; where an object is a shooting area. Pau, [0018] discloses “A minimum of three fringe patterns is needed for the reconstruction”; where three fringe patterns is at least two structured lights of different patterns);
first (Pau, [0018] discloses “In operation, the fringe pattern is projected onto an object, and the reflected images from the object are measured using a camera”; where a camera is an image sensor); “and
a processor connected to the projection device and the first, configured to determine the depth data of an object in the shooting area according to the set of image frame pairs obtained by imaging the structured light” (Pau, [0028] discloses “The components in the exemplary diagram of FIG. 4 include an electronic device such as a laptop 402 (including a processor and a memory, among other components) that is used to drive a projection device 404.” Pau, [0018] discloses “The depth of the object at each pixel is calculated from the phase, φ(k,l), where (k,l) denotes the index of the camera pixel”; where depth of an object is depth data of the object in the shooting area),
“wherein, a first sub- image sensor and a second sub-image sensor that share at least part of an optical path, (Pau, [0027] discloses “FIG. 3 illustrates a system 300 that can be used for 3D imaging of an object in accordance with an exemplary embodiment” and “The reflected light 310 is collected by collection optics 308 and detected by a sensor 303, such as an array sensor, typically a CCD or CMOS array with pixelated color filter for the wavelength(s) of interest”; where an array sensor is an array of multiple sensors, and thus contains at least a first and second sub-image sensors; where the reflected light shares an optical path to the array sensor),
Pau does not explicitly teach “first and second image sensors having a predetermined relative positional relationship for capturing the shooting area to obtain a set of image frame pairs illuminated by the set of structured light,” “a processor connected to the projection device and the firstand second image sensors,” “wherein, each of the first and second image sensors comprises at least a first sub- image sensor and a second sub-image sensor that share at least part of an optical path, the first sub-image sensors of the first and second image sensors form a first sub-image sensor pair used to image the structured light of a first pattern among the different patterns,” and “and the second sub-image sensors of the first and second image sensors form a second sub-image sensor pair used to image the structured light of a second pattern among the different patterns after the projection of the first pattern” (emphasis added).
However, in an analogous field of endeavor, Mor teaches “first and second image sensors for capturing the shooting area to obtain a set of image frame pairs illuminated by the set of structured light” (Mor, [0055] discloses “Thus, for example, all of the sensing units simultaneously activate their respective stripes 142, followed by stripes 144, and so on, so that no more than a single sensing unit is active within each overlap region 140 at any given time. Each sensing unit provides 3D mapping data with respect to its own part of scene 130, and a processing unit (such as controller 121 or another computer) stitches the data together into a combined depth map”; where sensing units 124 and 126 are first and second image sensors; where parts of scene 130 are image frame pairs; where structured lights 142 and 144 are a set of structured light)
“a processor connected to the projection device and the first and second image sensors” (Mor, [0055] discloses “Each sensing unit provides 3D mapping data with respect to its own part of scene 130, and a processing unit (such as controller 121 or another computer) stitches the data together into a combined depth map”; where a controller 121 is a processor connected to first and second image sensors),
“wherein, each of the first and second image sensors comprises at least a first sub- image sensor and a second sub-image sensor that share at least part of an optical path, the first sub-image sensors of the first and second image sensors form a first sub-image sensor pair used to image the structured light of a first pattern among the different patterns” (Mor, [0055] discloses “Typically, the sensing units are controlled so that they illuminate and capture radiation from respective non-overlapping stripes 142, 144, 146, 148. Within each sensing unit, the illumination stripe and the sensing area that is triggered to receive radiation by the rolling shutter are internally synchronized as described above. Furthermore, the timing of all the sensing units is coordinated to avoid interference”; where non-overlapping stripes are different patterns; where sensing areas are sub-image sensors; where triggered “respective” stripes indicates stripes (patterns) are sensed by different sensing areas (sub-image sensors); see in Fig. 8 that both 124 and 126 illuminate and sense stripes 142, thus Mor teaches first and second image sensors forming a first sub-image pair of a first pattern)
“and the second sub-image sensors of the first and second image sensors form a second sub-image sensor pair used to image the structured light of a second pattern among the different patterns after the projection of the first pattern” (Mor, [0055] discloses “Typically, the sensing units are controlled so that they illuminate and capture radiation from respective non-overlapping stripes 142, 144, 146, 148. Within each sensing unit, the illumination stripe and the sensing area that is triggered to receive radiation by the rolling shutter are internally synchronized as described above. Furthermore, the timing of all the sensing units is coordinated to avoid interference”; where non-overlapping stripes are different patterns; where sensing areas are sub-image sensors; where triggered “respective” stripes indicates stripes (patterns) are sensed by different sensing areas (sub-image sensors); see in Fig. 8 that both 124 and 126 illuminate and sense stripes 144, thus Mor teaches first and second image sensors forming a second sub-image pair of a first pattern. Mor, [0055] discloses “Thus, for example, all of the sensing units simultaneously activate their respective stripes 142, followed by stripes 144, and so on, so that no more than a single sensing unit is active within each overlap region 140 at any given time”; where activating each stripe pattern simultaneously followed by the next stripe pattern is imaging the second pattern “after” the projection of the first pattern).
PNG
media_image1.png
495
549
media_image1.png
Greyscale
Fig. 8 of Mor
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified Pau to incorporate the teachings of Mor by using multiple sensing units and multiple stripe patterns and syncing acquisition. The prior art contained a ‘base’ device upon which the claimed invention can be seen as an ‘improvement.’ That is, Pau teaches determining depth data from multiple light pattern pulses at different wavelengths collected simultaneously. The prior art contained a ‘comparable’ device that has been improved in the same way as the claimed invention. That is, Mor teaches using light pattern stripes of different patterns, collected sequentially and by multiple sensors and multiple sensing areas. One of ordinary skill in the art could have applied the known ‘improvement’ technique in the same way to the ‘base’ device and the results would have been predictable to one of ordinary skill in the art. Additionally, one of ordinary skill in the art would be motivated to combine the Pau and Mor references in order to improve signal of a depth detecting system in the presence of bright lighting conditions: Mor, [0027] discloses “Systems based on projection of patterned light may suffer from low signal/background ratio due to limitations on the power of the projector, particularly in conditions of strong ambient light. Embodiments of the present invention address this problem by projecting radiation onto the scene of interest in a synchronized spatial sweep, which is timed to take advantage of the rolling shutter of the image sensor in order to improve the signal/background ratio of the system.”
Although Fig. 8 of Mor shows a known overlap region 140 between image sensors, the combination of Pau and Mor does not explicitly teach “first and second image sensors having a predetermined relative positional relationship” (emphasis added).
However, in an analogous field of endeavor, Yang teaches “first and second image sensors having a predetermined relative positional relationship” (Yang, [0011] also discloses “Preferably, the geometric calibration further through following method: wherein one image sensor as standard. left of the sensing surface of the pixel coordinate of the other image sensor so that the same object appear at the same position of all the sensors, wherein, M1 is the positional relationship of the image sensor calibration plate as the standard, M2 is the positional relationship of the calibration plate and the other image sensor”; where a defined M1 and M2 is a predetermined relative positional relationship between sensors).
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified Pau and Mor to incorporate the teachings of Yang by determining a positional relationship between an image sensor and an other image sensor. The prior art contained a ‘base’ device upon which the claimed invention can be seen as an ‘improvement.’ That is, Mor teaches determining depth data from multiple light pattern pulses collected from different image sensors with overlap regions. The prior art contained a ‘comparable’ device that has been improved in the same way as the claimed invention. That is, Yang teaches determining a positional relationship between image sensors. One of ordinary skill in the art could have applied the known ‘improvement’ technique in the same way to the ‘base’ device and the results would have been predictable to one of ordinary skill in the art. That is, it would have been obvious to one of ordinary skill in the art that applying the technique of Yang of determining positional relationships M1 and M2 between sensors would be applicable to the method of Mor where multiple sensors are implemented with overlap regions. Additionally, one of ordinary skill in the art would be motivated to combine the Pau and Yang references in order to increase picture frequency with low cost (Yang, abstract discloses “The invention can obviously improve the picture frequency and ensures the accuracy of the picture, and the device has simple structure and low cost.”) Accordingly, the combination of Pau, Mor, and Yang discloses the invention of Claim 1.
Regarding Claim 3, the combination of Pau, Mor, and Yang teaches “The depth data computing device according to claim 1, comprising:
synchronization device for making the first sub-image sensor pair and the second sub-image sensor pair to sequentially image the at least two structured light of different patterns, wherein the first sub-image sensor pair images simultaneously, the second sub-image pair images simultaneously, with a first interval therebetween, the at a first interval being smaller than a frame imaging interval of the sub-image sensor, while the projection device projects the structured light of the first pattern and the second pattern with the first interval therebetween, and making each of the first and second sub-image sensor perform its next frame imaging at a second interval not smaller than the frame imaging interval of the sub-image sensor, and is synchronized with the projection of the projection device” (Mor, [0055] discloses “Thus, for example, all of the sensing units simultaneously activate their respective stripes 142, followed by stripes 144, and so on, so that no more than a single sensing unit is active within each overlap region 140 at any given time”). The proposed combination as well as the motivation for combining the Pau, Mor, and Yang references presented in the rejection of Claim 1, apply to Claim 3 and are incorporated herein by reference. Thus, the apparatus recited in Claim 3 is met by Pau, Mor, and Yang.
Regarding Claim 5, the combination of Pau, Mor, and Yang teaches “The depth data computing device according to claim 1, wherein each of the first and second image sensors comprises:
lens assembly for receiving incident return structured light” (Mor, [0045] discloses “In such embodiments, the beams emitted by the different optoelectronic elements use different parts of lens 76”);
a beam splitting device for splitting the incident return structured light into at least a first beam and a second beam” (Mor, [0045] discloses “In such embodiments, the beams emitted by the different optoelectronic elements use different parts of lens 76, which may therefore be designed so that the collimated beams exit at respective angles corresponding to the desired vertical fan-out”; where beams using different parts of a lens to exit at different angles is a beam splitting device),
“wherein the first sub-image sensor images the first light beam corresponding to the returning structured light with the first pattern” (Mor, [0055] discloses “Typically, the sensing units are controlled so that they illuminate and capture radiation from respective non-overlapping stripes 142, 144, 146, 148. Within each sensing unit, the illumination stripe and the sensing area that is triggered to receive radiation by the rolling shutter are internally synchronized as described above”; see in Fig. 8 that both 124 and 126 illuminate and sense stripes 142, thus Mor teaches first and second image sensors forming a first sub-image pair of a first pattern and only receives radiation by rolling shutters for a first pattern 142); “and
the second sub-image sensor images the second light beams corresponding to the returning structured light with the second pattern” (Mor, [0055] discloses “Typically, the sensing units are controlled so that they illuminate and capture radiation from respective non-overlapping stripes 142, 144, 146, 148. Within each sensing unit, the illumination stripe and the sensing area that is triggered to receive radiation by the rolling shutter are internally synchronized as described above”; see in Fig. 8 that both 124 and 126 illuminate and sense stripes 144, thus Mor teaches first and second image sensors forming a second sub-image pair of a second pattern and only receives radiation by rolling shutters for a second pattern 144). The proposed combination as well as the motivation for combining the Pau, Mor, and Yang references presented in the rejection of Claim 1, apply to Claim 5 and are incorporated herein by reference. Thus, the apparatus recited in Claim 5 is met by Pau, Mor, and Yang.
Regarding Claim 6, the combination of Pau, Mor, and Yang teaches “The depth computing device according to claim 1, wherein each of the first and second image sensors comprises:
lens assembly for receiving incident return structured light” (Yang, [0020] discloses “a lens 1”);
an optical path conversion device for delivering the incident return structured light to at least a first sub-path and a second sub-path” (Yang, [0020] discloses “a half prism 2”; where a half prism is an optical path conversion device; where diverging paths in Fig. 1 of Yang are first and second sub-paths),
wherein the first sub-image sensor images the returning structured light corresponding to the first pattern on the first sub- path” (Mor, [0055] discloses “Within each sensing unit, the illumination stripe and the sensing area that is triggered to receive radiation by the rolling shutter are internally synchronized as described above. Furthermore, the timing of all the sensing units is coordinated to avoid interference.”), “and
the second sub-image sensor images the returned structured light corresponding to the second different pattern on the second sub-path” (Yang, [0020] discloses “a second image sensor 5”). The proposed combination as well as the motivation for combining the Pau, Mor, and Yang references presented in the rejection of Claim 1, apply to Claim 6 and are incorporated herein by reference. Thus, the apparatus recited in Claim 6 is met by Pau, Mor, and Yang.
Regarding Claim 7, the combination of Pau, Mor, and Yang teaches “The depth data computing device according to claim 1, wherein in each of the first and second image sensors, the first sub-image sensor and the second sub-image sensor that share at least part of the optical path have the same optical path length” (Yang, see Fig. 1, where the path between the lens and prism is shared). The proposed combination as well as the motivation for combining the Pau, Mor, and Yang references presented in the rejection of Claim 1, apply to Claim 7 and are incorporated herein by reference. Thus, the apparatus recited in Claim 7 is met by Pau, Mor, and Yang.
Regarding Claim 8, the combination of Pau, Mor, and Yang teaches “The depth data computing device according to claim 7, wherein in each of the first and second image sensors, the first sub-image sensors and the second sub-image sensor that share at least part of the optical path are aligned at the pixel level” (Yang, [0011] also discloses “Preferably, the geometric calibration further through following method: wherein one image sensor as standard. left of the sensing surface of the pixel coordinate of the other image sensor so that the same object appear at the same position of all the sensors, wherein, M1 is the positional relationship of the image sensor calibration plate as the standard, M2 is the positional relationship of the calibration plate and the other image sensor”). The proposed combination as well as the motivation for combining the Pau, Mor, and Yang references presented in the rejection of Claim 1, apply to Claim 8 and are incorporated herein by reference. Thus, the apparatus recited in Claim 8 is met by Pau, Mor, and Yang.
Regarding Claim 9, the combination of Pau, Mor, and Yang teaches “The depth data computing device according to claim 1, wherein each sub-image sensor is an infrared light sensors” (Pau, [0026] discloses “For many applications, the wavelengths can be near infrared color bands, where the projected light is invisible to the human eye.” Pau, [0027] discloses “The reflected light 310 is collected by collection optics 308 and detected by a sensor 303, such as an array sensor, typically a CCD or CMOS array with pixelated color filter for the wavelength(s) of interest”; where a sensor for wavelengths of interest, the wavelengths being near infrared color bands, is infrared light sensors).
Regarding Claim 10, the combination of Pau, Mor, and Yang teaches “The depth data computing device according to claim 1, wherein the set of structured lights with different patterns projected by the projection device is a set of structured lights with different coded stripes” (Pau, [0028] discloses “The camera module 406, which is sensitive to different wavelengths, processes the collected images (alone or in cooperation with the laptop 402), and produces three intensity values associated with fringe patterns 408, 409 and 410 (each corresponding to a particular wavelength or range of wavelengths)”; where different fringe patterns are different coded stripes; see Fig. 4).
PNG
media_image2.png
544
691
media_image2.png
Greyscale
Fig. 4 of Pau
Regarding Claim 16, Pau teaches “A method for measuring depth data” (Pau, [0016] discloses “The disclosed embodiments relate to methods, devices and systems that apply digital fringe projection (DFP) techniques to facilitate the generation of three-dimensional (3D) images of an object based on the measurement of polarizations and/or colored light in a single shot”), “comprising:
scanning and projecting structured light to a shooting area” (Pau, [0017] discloses “DFP methods typically utilize a projector, such as a digital light processing (DLP), liquid crystal display (LCD) or a liquid crystal on silicon (LCoS) projector, to project computer generated fringe patterns onto an object”; where computer generated fringe patterns are structured light having different patterns; where an object is a shooting area);
“capturing the shooting area to obtain a first image frame first structured light of a first pattern by using a first sub-image sensor (Pau, [0018] discloses “In operation, the fringe pattern is projected onto an object, and the reflected images from the object are measured using a camera”; where a camera is an image sensor; where a fringe pattern is a first pattern);
“ scanning and projecting second structured light of a second pattern to the shooting area, the second pattern is different from the first pattern” (Pau, [0018] discloses “A minimum of three fringe patterns is needed for the reconstruction”; where three fringe patterns is at least two structured lights of different patterns);
“capturing the shooting area to obtain a second image frame sub-image sensor of the first sub-image sensor pair and one sub-image sensor of the second sub- image sensor pair share at least part of an optical path and form a first image sensor” (Pau, [0027] discloses “FIG. 3 illustrates a system 300 that can be used for 3D imaging of an object in accordance with an exemplary embodiment” and “The reflected light 310 is collected by collection optics 308 and detected by a sensor 303, such as an array sensor, typically a CCD or CMOS array with pixelated color filter for the wavelength(s) of interest”; where an array sensor is an array of multiple sensors, and thus contains at least two sub-image sensors; where the reflected light shares an optical path to the array sensor), and
determining the depth data of the object to be measured in the shooting area according to the first and second image frame pairs” (Pau, [0018] discloses “The depth of the object at each pixel is calculated from the phase, φ(k,l), where (k,l) denotes the index of the camera pixel”; where depth of an object is depth data of the object in the shooting area).
Pau does not explicitly teach “a first image frame pair illuminated by structured light by using a first sub-image sensor pair with a predetermined relative positional relationship,” “after the projection of the first structured light, scanning and projecting second structured light of a second pattern,” “a second image frame pair under the illumination of the second structured light by using a second sub-image sensor pair with a predetermined relative positional relationship,” and “the other sub-image sensor of the first sub-image sensor pair and the other sub-image sensor of the second sub-image sensor pair share at least part of an optical path and form a second image sensor.”
However, in an analogous field of endeavor, Mor teaches “a first image frame pair illuminated by structured light by using a first sub-image sensor pair ” (Mor, [0055] discloses “Typically, the sensing units are controlled so that they illuminate and capture radiation from respective non-overlapping stripes 142, 144, 146, 148. Within each sensing unit, the illumination stripe and the sensing area that is triggered to receive radiation by the rolling shutter are internally synchronized as described above. Furthermore, the timing of all the sensing units is coordinated to avoid interference”; where non-overlapping stripes are structured light; where sensing areas are sub-image sensors; where triggered “respective” stripes indicates stripes (patterns) are sensed by different sensing areas (sub-image sensors); see in Fig. 8 that both 124 and 126 illuminate and sense stripes 142, thus Mor teaches first and second image sensors forming a first sub-image pair of a first pattern),
“after the projection of the first structured light, scanning and projecting second structured light of a second pattern,” (Mor, [0055] discloses “Thus, for example, all of the sensing units simultaneously activate their respective stripes 142, followed by stripes 144, and so on, so that no more than a single sensing unit is active within each overlap region 140 at any given time”; where activating each stripe pattern simultaneously followed by the next stripe pattern is imaging the second pattern “after” the projection of the first structured light),
“a second image frame pair under the illumination of the second structured light by using a second sub-image sensor pair (Mor, [0055] discloses “Typically, the sensing units are controlled so that they illuminate and capture radiation from respective non-overlapping stripes 142, 144, 146, 148. Within each sensing unit, the illumination stripe and the sensing area that is triggered to receive radiation by the rolling shutter are internally synchronized as described above. Furthermore, the timing of all the sensing units is coordinated to avoid interference”; where non-overlapping stripes are different patterns; where sensing areas are sub-image sensors; where triggered “respective” stripes indicates stripes (patterns) are sensed by different sensing areas (sub-image sensors); see in Fig. 8 that both 124 and 126 illuminate and sense stripes 144, thus Mor teaches first and second image sensors forming a second sub-image sensor pair), and
“the other sub-image sensor of the first sub-image sensor pair and the other sub-image sensor of the second sub-image sensor pair share at least part of an optical path and form a second image sensor” (Mor, [0055] discloses “Thus, for example, all of the sensing units simultaneously activate their respective stripes 142, followed by stripes 144, and so on, so that no more than a single sensing unit is active within each overlap region 140 at any given time. Each sensing unit provides 3D mapping data with respect to its own part of scene 130, and a processing unit (such as controller 121 or another computer) stitches the data together into a combined depth map”; where sensing units 124 and 126 are first and second image sensors; see Fig. 8 where sensing unit 126 contains stripes 142 and 144 which share an optical path).
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified Pau to incorporate the teachings of Mor by using multiple sensing units and multiple stripe patterns and syncing acquisition. The prior art contained a ‘base’ device upon which the claimed invention can be seen as an ‘improvement.’ That is, Pau teaches determining depth data from multiple light pattern pulses at different wavelengths collected simultaneously. The prior art contained a ‘comparable’ device that has been improved in the same way as the claimed invention. That is, Mor teaches using light pattern stripes of different patterns, collected sequentially and by multiple sensors and multiple sensing areas. One of ordinary skill in the art could have applied the known ‘improvement’ technique in the same way to the ‘base’ device and the results would have been predictable to one of ordinary skill in the art. Additionally, one of ordinary skill in the art would be motivated to combine the Pau and Mor references in order to improve signal of a depth detecting system in the presence of bright lighting conditions: Mor, [0027] discloses “Systems based on projection of patterned light may suffer from low signal/background ratio due to limitations on the power of the projector, particularly in conditions of strong ambient light. Embodiments of the present invention address this problem by projecting radiation onto the scene of interest in a synchronized spatial sweep, which is timed to take advantage of the rolling shutter of the image sensor in order to improve the signal/background ratio of the system.”
Although Fig. 8 of Mor shows a known overlap region 140 between image sensors, the combination of Pau and Mor does not explicitly teach “a first sub-image sensor pair with a predetermined relative positional relationship” and “a second sub-image sensor pair with a predetermined relative positional relationship” (emphasis added).
However, in an analogous field of endeavor, Yang teaches teach “a first sub-image sensor pair with a predetermined relative positional relationship” and “a second sub-image sensor pair with a predetermined relative positional relationship” (Yang, [0011] also discloses “Preferably, the geometric calibration further through following method: wherein one image sensor as standard. left of the sensing surface of the pixel coordinate of the other image sensor so that the same object appear at the same position of all the sensors, wherein, M1 is the positional relationship of the image sensor calibration plate as the standard, M2 is the positional relationship of the calibration plate and the other image sensor”; where a defined M1 and M2 is a predetermined relative positional relationship between sensors).
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified Pau and Mor to incorporate the teachings of Yang by determining a positional relationship between an image sensor and an other image sensor. The prior art contained a ‘base’ device upon which the claimed invention can be seen as an ‘improvement.’ That is, Mor teaches determining depth data from multiple light pattern pulses collected from different image sensors with overlap regions. The prior art contained a ‘comparable’ device that has been improved in the same way as the claimed invention. That is, Yang teaches determining a positional relationship between image sensors. One of ordinary skill in the art could have applied the known ‘improvement’ technique in the same way to the ‘base’ device and the results would have been predictable to one of ordinary skill in the art. That is, it would have been obvious to one of ordinary skill in the art that applying the technique of Yang of determining positional relationships M1 and M2 between sensors would be applicable to the method of Mor where multiple sensors are implemented with overlap regions. Additionally, one of ordinary skill in the art would be motivated to combine the Pau and Yang references in order to increase picture frequency with low cost (Yang, abstract discloses “The invention can obviously improve the picture frequency and ensures the accuracy of the picture, and the device has simple structure and low cost.”) Accordingly, the combination of Pau, Mor, and Yang discloses the invention of Claim 16.
Regarding Claim 18, the combination of Pau, Mor, and Yang teaches “The method according to claim 16, wherein scanning and projecting second structured light of the second pattern to the shooting area comprises:
projecting the second structured light after the projection of the first structured light, with a first interval therebetween, the at a first interval being smaller than a frame imaging interval of the sub-image sensor, and
time interval between capturing the first image frame pair by the first sub-image sensor pair and capturing the second image frame pair by the second sub-image sensor pair is smaller than the frame imaging interval of the sub-image sensors” (Yang, Fig. 2 and [0023] discloses “C1, C2 are the first image sensor 3 and second image sensor 5 exposure timing graph, C is the exposure timing graph of a single image sensor equivalent. time sequence control is as follows: the controller 4 controls the first image sensor 3 exposure, the time, the controller 4 controls the second image sensor 5 exposure, then T 2 elapsed time, the controller 4 controls the first image sensor 3 exposure again, repeating the above process, so the picture frequency is increased to two times of single image sensor, achieving frequency acquisition of the image”). The proposed combination as well as the motivation for combining the Pau, Mor, and Yang references presented in the rejection of Claim 16, apply to Claim 18 and are incorporated herein by reference. Thus, the apparatus recited in Claim 18 is met by Pau, Mor, and Yang.
Regarding Claim 19, the combination of Pau, Mor, and Yang teaches “The method according to claim 18, further comprising:
projecting a third structured light of a third pattern to the shooting area after the projection of the second structured light, with a second interval therebetween, the second interval being not smaller than the frame imaging interval of the sub-image sensors, the third pattern is different from the first pattern and the second pattern” (Pau, [0018] discloses “A minimum of three fringe patterns is needed for the reconstruction.” Mor also shows in Fig. 8 pattern 146);
capturing the shooting area to obtain a third image frame pair under the illumination of the third structured light by using the first sub-image sensor pair, wherein the third image frame pair is used to determine the depth data of the shooting area” (Pau, [0018] discloses “The depth of the object at each pixel is calculated from the phase, φ(k,l), where (k,l) denotes the index of the camera pixel”). The proposed combination as well as the motivation for combining the Pau, Mor, and Yang references presented in the rejection of Claim 16, apply to Claim 19 and are incorporated herein by reference. Thus, the apparatus recited in Claim 19 is met by Pau, Mor, and Yang.
Regarding Claim 21, the combination of Pau, Mor, and Yang teaches “The method according to claim 16, wherein in each of the first and second image sensors, the first sub- image sensor and the second sub-image sensor respectively acquire the split beams of the first structured light and the second structured light, and selectively turning on one of the first sub-image sensor and the second sub-image sensor for capturing” (Mor, [0045] discloses “In such embodiments, the beams emitted by the different optoelectronic elements use different parts of lens 76, which may therefore be designed so that the collimated beams exit at respective angles corresponding to the desired vertical fan-out”; where beams using different parts of a lens to exit at different angles is a beam splitting device. Mor, [0055] discloses “Typically, the sensing units are controlled so that they illuminate and capture radiation from respective non-overlapping stripes 142, 144, 146, 148. Within each sensing unit, the illumination stripe and the sensing area that is triggered to receive radiation by the rolling shutter are internally synchronized as described above”; see in Fig. 8 that both 124 and 126 illuminate and sense stripes 142, thus Mor teaches first and second image sensors forming a first sub-image pair of a first pattern and only receives radiation by rolling shutters for a first pattern 142; see in Fig. 8 that both 124 and 126 illuminate and sense stripes 144, thus Mor teaches first and second image sensors forming a second sub-image pair of a second pattern and only receives radiation by rolling shutters for a second pattern 144). The proposed combination as well as the motivation for combining the Pau, Mor, and Yang references presented in the rejection of Claim 16, apply to Claim 21 and are incorporated herein by reference. Thus, the method recited in Claim 21 is met by Pau, Mor, and Yang.
Regarding Claim 23, the combination of Pau, Mor, and Yang teaches “The method according to claim 16, wherein the optical path of the incident light is controlled so that in each of the first and second image sensors, only the first sub-image sensor acquires and captures the first structured light, and only the second sub-image sensor acquires and captures the second structured light” (Yang, Fig. 1; see first sensor 3 and second sensor 5 only acquire light from their respective optical paths; see also Fig. 8 of Mor). The proposed combination as well as the motivation for combining the Pau, Mor, and Yang references presented in the rejection of Claim 16, apply to Claim 23 and are incorporated herein by reference. Thus, the method recited in Claim 23 is met by Pau, Mor, and Yang.
Regarding Claim 24, the combination of Pau, Mor, and Yang teaches “The method according to claim 18, further comprising:
projecting a third structured light of a third pattern to the shooting area after the projection of the second structured light, with the first interval therebetween, the third pattern is different from the first pattern and the second pattern” ” (Pau, [0018] discloses “A minimum of three fringe patterns is needed for the reconstruction.” Mor also shows in Fig. 8 pattern 146); “and
capturing the shooting area to obtain a third image frame pair under the illumination of the third structured light by using a third sub-image sensor pair, wherein one sub-image sensor of the third sub-image sensor pair belongs to the first image sensor and shares at least part of the optical path with other sub-image sensors of the first image sensor, the other sub-image sensor of the third sub-image sensor pair belongs to the second image sensor and shares at least part of the optical path with other sub-image sensors of the second image sensor, the third image frame pair is used to determine the depth data of the shooting area” (Pau, [0018] discloses “The depth of the object at each pixel is calculated from the phase, φ(k,l), where (k,l) denotes the index of the camera pixel”; see also Fig. 8 of Mor where optical paths of patterns 142, 144, and 146 share optical paths for each sensor (bordering patterns share optical path)). The proposed combination as well as the motivation for combining the Pau, Mor, and Yang references presented in the rejection of Claim 16, apply to Claim 24 and are incorporated herein by reference. Thus, the method recited in Claim 24 is met by Pau, Mor, and Yang.
Claims 11-13 are rejected under 35 U.S.C. 103 as being unpatentable over Pau (US 2021/0183085 A1), in view of Mor (US 2019/0068951 A1), further in view of Yang et al. (CN 103780844 A), further in view of Wang et al. (CN 209927097 U).
Regarding Claim 11, the combination of Pau, Mor, and Yang teaches “The depth data computing device according to claim 1, wherein the projection device comprises:
a laser generator for generating line-shaped and/or point laser light” (Pau, [0026] discloses “The near infrared wavelengths can be closely spaced, such that the same type of semiconductor laser can be used to provide the projected fringes”),
The combination of Pau, Mor, and Yang does not explicitly teach “the laser generator performs high-speed switching to scan and project light and dark structured light corresponding to the stripe code.”
However, in an analogous field of endeavor, Wang teaches “the laser generator performs high-speed switching to scan and project light and dark structured light corresponding to the stripe code” (Wang, page 2, paragraph 6 discloses “Preferably, the projection device may include a laser generator for generating linear and/or infrared laser, and the laser generator for high speed switching to scanning the structure light bright-dark projection corresponding to the stripe code alternately”).
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified the combination of Pau, Mor, and Yang to incorporate the teachings of Wang by implementing a high speed switching laser to scan the structured light. One of ordinary skill in the art would be motivated to combine the Pau, Mor, Yang, and Wang references in order to control coding patterns precisely: Wang, page 2, paragraph 7 discloses “Thus, by simply laser generator switch to realize precise control of the coding pattern.” Accordingly, the combination of the Pau, Mor, Yang, and Wang references discloses the invention of Claim 11.
Regarding Claim 12, the combination of Pau, Mor, Yang, and Wang teaches “The depth data computing device according to claim 11, wherein the projection device comprises:
a light emitting device for generating line-shaped light” (Wang, page 5, paragraph 4 discloses “In one embodiment, the laser generator 411 may be linear laser generator to generate extending in the x-direction of linear light (in FIG. 4A-B perpendicular to the direction of the paper surface)); “and
a reflecting device for reflecting the line-shaped light to project the line-shaped light moving in a direction perpendicular to the stripe direction to the shooting area” (Wang, page 5, paragraph 4 discloses “the linear light followed by swing can move up and down along the x direction of the shaft of the projection mechanism 412 (also may be referred to as a ‘reflecting mechanism’) projected to the imaging plane”). The proposed combination as well as the motivation for combining the Pau, Mor, Yang, and Wang references presented in the rejection of Claim 11, apply to Claim 12 and are incorporated herein by reference. Thus, the apparatus recited in Claim 12 is met by Pau, Mor, Yang, and Wang.
Regarding Claim 13, the combination of Pau, Mor, Yang, and Wang teaches “The depth data computing device according to claim 11, wherein the each sub-image sensor is a global image sensor” (Yang, [0010] “the same object appear at the same position of all the sensors; the brightness calibration is as follows: control simultaneously expose two image sensors, two pieces image shoot the same object at the same time, wherein one is standard”; where, as best understood in light of the specification, a global image sensor is a sensor where all pixels are imaged simultaneously).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Pau (US 2021/0183085 A1), in view of Mor (US 2019/0068951 A1), further in view of Yang et al. (CN 103780844 A), further in view of Wang et al. (CN 209927097 U), further in view of Campbell (US 9204041 B1).
Regarding Claim 14, the combination of Pau, Mor, Yang, and Wang does not explicitly teach the device of Claim 14.
However, in an analogous field of endeavor, Campbell teaches “The depth data computing device according to claim 13, wherein in each of the first and second image sensors, the first sub-image sensor and the second sub-image sensor that share at least part of the optical path are installed upside down from each other” (Campbell, Claim 1 discloses “a plurality of cameras, each camera having a rolling shutter for enabling the capture of panoramic image data from a field of view of the camera; a first camera pair comprising a first camera positioned opposite a second camera, the first camera and the second camera oriented to face outwards in substantially opposite directions”).
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified the combination of Pau, Mor, Yang, and Wang to incorporate the teachings of Campbell by positioning rolling shutters opposite each other. One of ordinary skill in the art would be motivated to combine the Pau, Mor, Yang, Wang, and Campbell references in order to reduce time-delay artifacts: Campbell, column 3, lines 5-9 discloses “The opposing and converging RSs capture image data at the cameras' overlapping FOV at temporally proximate times, thus preventing or significantly reducing time-delay artifacts.” Accordingly, the combination of Pau, Mor, Yang, Wang, and Campbell discloses the invention of Claim 14.
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Pau (US 2021/0183085 A1), in view of Mor (US 2019/0068951 A1), further in view of Yang et al. (CN 103780844 A), further in view of Wang et al. (CN 209927097 U), further in view of Toyoda (JP 2008224629 A).
Regarding Claim 26, the combination of Pau, Mor, Yang, and Wang does not explicitly teach the device of Claim 26.
However, in an analogous field of endeavor, Toyoda teaches “The depth data computing device according to claim 11, wherein each two sub-image sensor is a rolling shutter image sensors” (Toyoda, page 5, paragraph 1 discloses “There is provided a rolling shutter device 9 (rolling shutter means) that reads two single-eye images A and B with a time difference in the order of single-eye images A and B within one shutter operation”) “and the depth data computing device further comprises:
a column synchronization device, for synchronously enabling the pixel column in the stripe direction corresponding to the current scanning position in the sub-image sensor currently used for imaging to perform imaging based on the scanning position of the projection device” (Toyoda, page 5, paragraph 1 discloses “A slit pattern switching signal s <b> 2 is output, and the slit pattern switching timing of the pattern irradiating device 2 is controlled to be synchronized with the reading timing of the single-eye image from the compound-eye imaging device 3.”)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to have modified the combination of Pau, Yang, and Wang to incorporate the teachings of Toyoda by implementing synchronize pattern switching timing and a rolling shutter device. One of ordinary skill in the art would be motivated to combine the Pau, Yang, Wang, and Toyoda references in order to reduce time of imaging: Toyoda, page 4, paragraph 2 discloses “Since the plurality of slit patterns are sequentially switched and irradiated, a plurality of images respectively corresponding to the plurality of slit patterns can be obtained by one imaging operation, and the time required for imaging can be shortened.” Accordingly, the combination of Pau, Yang, Wang, and Toyoda discloses the invention of Claim 26.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAROLINE TABANCAY DUFFY whose telephone number is (703)756-1859. The examiner can normally be reached Monday - Friday 8:00 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at 5712723382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAROLINE TABANCAY DUFFY/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights