DETAILED ACTION
Allowable Subject Matter
Claim 7-8 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-14 have been considered but are moot because the new grounds of rejection.
Regarding to the 35 U.S.C 112(f) interpretation of claims 13-14, the amendment has cured the basis of the 35 U.S.C 112(f) interpretation. Therefore, the 35 U.S.C 112(f) interpretation of claim 13-14 is hereby withdrawn.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Doepke (US 20120293610 A1) in view of Veldandi (US 20180182143 A1) and in further view of Henningsson (US 20180367789 A1).
As per claim 1, Doepke teaches the claimed:
1. A method of generating stitched image data for an application,
defining different interest levels for image data of different spatial regions or of different color components of a color space depending on an application for which the stitched image data is generated,
wherein the different levels correspond to different levels of interest of matter depicted by the image data for an end application or an end user of the generated stitched images,
(Doepke [0025]: “identifying a feature of interest that is represented at a location in each of the first image and the second image, wherein each of the representations are located in the overlapping region; selecting the representation of the feature of interest from the first; blending between the first image and the second image in the overlapping region to generate a resulting overlapping region; and assembling the first image and the second image, using the resulting overlapping region to replace the overlapping region between the first image and the second image, wherein the act of blending excludes the location of the identified feature of interest, and wherein the selected representation is used in the resulting overlapping region at the location of the identified feature of interest.”
Doepke teaches the images that share a feature of interest at a location in each of the separate images, which is the interest level for the image data of different spatial regions. Additionally, the different levels, which are the feature of interest location and where the feature of interest is not, correspond to interest levels for an end application as the final result of the processing results using the different regions of the images.)
dividing each set of image data into portions of different defined interest levels corresponding to a same plurality of spatial regions of the blending region or corresponding to a same plurality of color components,
(Doepke [0025]: “identifying a feature of interest that is represented at a location in each of the first image and the second image, wherein each of the representations are located in the overlapping region”
Doepke [0081]: “each image may be divided into a plurality of segments (Step 802). An image segment may be defined as a portion of an image of predetermined size. In addition to the image information, the process 206 may acquire metadata information, e.g., the positional information corresponding to the image frames to be registered (Step 804). Through the use of an image registration algorithm involving, e.g., a feature detection algorithm”.
Doepke teaches dividing images into a plurality of segments based on a feature detection algorithm, which corresponds to the same regions that will be blended.)
Doepke alone does not explicitly teach the remaining claim limitations.
However, Doepke in combination with Veldandi and Henningsson teaches the claimed:
for each portion of image data:
determining one or more image frequency bands for the portion in view of the interest level of the portion, and
(Doepke [0081]: “each image may be divided into a plurality of segments … Through the use of an image registration algorithm involving, e.g., a feature detection algorithm”. Doepke teaches the image segments based on the feature detection algorithm, which is the interest level of that segment.
Veldandi [0036]: “the configurable number of bands may be selected and/or identified”
Veldandi [0037]: “the bandwidth associated with the final band may be considered to be an optimal bandwidth.”
Veldandi teaches a panoramic stitching system that uses a configurable number of bands for the overlapping and seam portion, which determines the bandwidth of the final band. The configurable number of bands will be determined using the image segments that were previously determined by Doepke based on the interest level, which will determine the frequency bands for the image segments in view of the interest levels.)
obtaining image data of the determined one or more image frequency bands from the portion of image data, and
(Veldandi [0041]: “… after the bandwidth has been selected for the final band, the overlapping regions of the two adjacent images to be combined are divided into multiple laplacian images for each of the bands.”
Veldandi [0045]: “after the appropriate masks have been applied to the various bands to create blended bands, the blended bands may be upscaled and summed to provide a final seam area that can be added to a panoramic image and/or otherwise used in connection with the creation of a combined panoramic image.”)
wherein the image data is acquired by one or more image sensors arranged to acquire image data depicting at least partly overlapping views of a scene,
(Henningsson [0005-0006]: “an object of the present invention to provide an improved process for stitching together images being captured simultaneously by multiple sensors of a video camera. … the above object is achieved by a method performed in a multi-sensor video camera having a first and a second sensor with partly overlapping fields of view”.)
the method comprising:
obtaining a first set of image data and a second set of image data, wherein each set of image data represents a blending region,
(Henningsson [0013]: “wherein image data from the overlapping portion of the first video frame is blended with image data from the overlapping portion of the second video frame in at least one of the steps of preparing the overlay and adding the overlay.”
Henningsson [0019]: “the step of preparing the overlay includes blending image data from the overlapping portion of the second video frame with image data from the overlapping portion of the first video frame.”)
blending the first set of image data and the second set of image data by multi-band blending,
(Henningsson [0039]: “The step of calculating the mask based on the image data from the overlapping portions of the second and the first video frame thus corresponds to calculating the weights associated with image data from the second and first video frames. … the mask may be calculated so as to include weights associated with multi-band blending”.
Henningsson teaches the weights that are associated with the image data of the second and first vides frames and are used in the calculation with the multi-band blending.)
wherein only the obtained image data of the determined one or more image frequency bands are blended for each portion of image data, thereby
(Veldandi [0041]: “… after the bandwidth has been selected for the final band, the overlapping regions of the two adjacent images to be combined are divided into multiple laplacian images for each of the bands.”
Veldandi [0045]: “after the appropriate masks have been applied to the various bands to create blended bands, the blended bands may be upscaled and summed to provide a final seam area that can be added to a panoramic image and/or otherwise used in connection with the creation of a combined panoramic image.”)
generating stitched image data for the blending region for the application.
(Henningsson [0069]: “the processing pipeline 104 is configured to stitch the video frames 108a, 108b together to create a panorama image 110 of the scene. The panorama image 110 may thus correspond to the combined, i.e., the union of the, fields of view of the image sensors 102a, 102.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the image bands as taught by Veldandi with the system of Doepke in order to blend different levels of image detail separately, reducing noticeable seams while still preserving important details. Also to use the stitching of overlapping images as taught by Hessingsson with the system of Doepke in order to create an image that blends and aligns the common ground between the overlapping images.
As per claims 12 and 13, these claims are similar in scope to limitations recited in claim 1, and thus is rejected under the same rationale.
As per claim 2, Doepke and Veldandi teaches the claimed:
2. The method according to claim 1, wherein the step of determining one or more image frequency bands comprises selecting one or more image frequency bands from a fixed set of image frequency bands.
(Veldandi [0013]: “In some example implementations of such a computer program product the indication of the plurality of bands is a predetermined number of bands.”
Veldandi [0036]: “the configurable number of bands may be selected and/or identified”.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the image bands as taught by Veldandi with the system of Doepke in order to blend different levels of image detail separately, reducing noticeable seams while still preserving important details.
Claims 3-5 are rejected under 35 U.S.C. 103 as being unpatentable over Doepke in view of Veldandi in view of Henningsson and in further view of Traub (“Loci-Segmented: Improving Scene Segmentation Learning”, 2023).
As per claim 3, Doepke, Veldandi and Henningsson alone does not explicitly teach the claimed limitations.
However, Doepke, Veldandi, and Henningsson in combination with Traub teaches the claimed:
3. The method according to claim 1, wherein the sets of image data are divided into portions corresponding to a same plurality of spatial regions of the blending region, and comprises:
performing image segmentation on the sets of image data, and dividing the sets of image data into spatial portions corresponding to segmented regions of different defined interest levels.
(Traub (Abstract): “Loci-Segmented (Loci-s), an advanced scene segmentation neural network”. Traub teaches the image segmentation of a scene.
Traub (Sec 2): “Each slot k in the slotted ResNet-based encoder module … the encoder produces Gestalt codes ˜ Gt k and positional codes ˜ Pt k, which includes object location (xk,yk), size (σk), and priority (ρk).” Traub teaches the slots that are spatial portions or regions of the scene, and also indicates different interest level with the priority level the encoder produces.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the scene segmentation as taught by Traub with the system of Doepke as modified by Veldandi and Henningsson in order to improve computational efficiency by allocating more resources to higher priority regions.
As per claim 4, Doepke, Veldandi, and Henningsson alone does not explicitly teach the claimed limitations.
However, Doepke, Veldandi, and Henningsson in combination with Traub teaches the claimed:
4. The method according to claim 3, wherein the image segmentation is performed using a background model.
(Traub (Sec 2.5): “is the development of a dedicated Background Module. … this module is principally bifurcated into two core elements: an Uncertainty Network and a Background Extraction component. … The nomenclature ’Uncertainty’ emanates from the inherently volatile nature of dynamic foreground objects, contrasting with the generally stable background elements in natural scenes.”
Traub (Sec 2.5): “The output from the Uncertainty Network serves as a masking function … By selectively masking out foreground objects, we introduce a bias favoring the exclusive reconstruction of background elements.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the scene segmentation as taught by Traub with the system of Doepke as modified by Veldandi and Henningsson in order to improve computational efficiency by allocating more resources to higher priority regions.
As per claim 5, Doepke, Veldandi and Henningsson alone does not explicitly teach the claimed limitations.
However, Doepke, Veldandi, and Henningsson in combination with Traub teaches the claimed:
5. The method according to claim 3, wherein the image segmentation is performed using an image classification algorithm or image recognition algorithm.
(Traub (Abstract): “Loci-Segmented (Loci-s), an advanced scene segmentation neural network”. Traub teaches the image segmentation that uses neural networks, which inherently perform classification or recognitions.
Traub (Sec 2): “Each slot k in the slotted ResNet-based encoder module … the encoder produces Gestalt codes ˜ Gt k and positional codes ˜ Pt k, which includes object location (xk,yk), size (σk), and priority (ρk).” Traub teaches the encoder that is able to determine the object location and size, which requires some kind of recognition.
Traub (Sec 2.5): “a masking function for input patches directed toward a masked autoencoder, which is modeled using a Vision Transformer architecture.” Traub teaches the Vision Transformers which are image classification models.
Traub (Sec 2.7): “the deployment of a specialized segmentation network akin to YOLACT (Bolya et al., 2019). Initial slot positions are calculated based on the instance masks outputted by this network.” Traub teaches TOLACT, which is a real time instance segmentation and classification network.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the scene segmentation as taught by Traub with the system of Doepke as modified by Veldandi and Henningsson in order to improve computational efficiency by allocating more resources to higher priority regions.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Doepke in view of Veldandi in view of Henningsson and in further view of Kim (US 20080095436 A1).
As per claim 6, Doepke, Veldandi, and Henningsson alone does not explicitly teach the claimed limitations.
However, Doepke, Veldandi, and Henningsson in combination with Kim teaches the claimed:
6. The method according to claim 1, wherein the sets of image data are divided into portions corresponding to a same plurality of spatial regions of the blending region, the method further comprising:
performing motion analysis on the sets of image data
(Kim [0054]: “The image motion detection unit 312 may detect motion of the image based on difference information received from the frame calculation unit 311. The camera motion detection unit 313 may detect motion of the camera taking the image, based on the difference information received from the frame calculation unit 311.”
Kim [0055]: “The foreground object detection unit 320 may detect an independently moving foreground object in the image.”)
and dividing the sets of image data into spatial regions
(Kim [0006]: “a foreground and a background in a video can be segmented from each other”
Kim [0040]: “segment the image signal into a foreground region and a background region according to the received image analysis signal.”
Kim [0079]: “Feature points may be segmented into those included in a foreground region and those included in a background region from the feature point segmentation result.”)
corresponding to different motion levels
(Kim [0044]: “since an image is taken by a still camera, a background does not move and only a foreground object moves. Therefore, the foreground region and the background region may be effectively segmented from each other using, for example, a background subtraction method.”
Kim [0074]: “Thus, whether an independently moving object exists or not may be determined by the number of outliers 820 that are not consistent with the motion of the camera”)
having different defined interest levels.
(Kim [0006]: “The foreground is important and may be in the center portion of a video screen. The foreground may include objects in good focus or near a camera, e.g., a key person or object in the screen. The background is the remaining part of the screen excluding the foreground and may be of lesser importance or no interest”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the motion analysis as taught by Kim with the system of Doepke as modified by Veldandi and Henningsson in order to improve computational efficiency by allocating more resources to higher priority regions that incorporate more movement within the image data.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Doepke in view of Veldandi in view of Henningsson and in further view of Park (“Dual-Color Space Network with Global Priors for Photo Retouching”, 2023).
As per claim 9, Doepke, Veldandi, and Henningsson alone does not explicitly teach the claimed limitations.
However, Doepke, Veldandi, and Henningsson in combination with Park teaches the claimed:
9. The method according to claim 1, wherein the sets of image data are divided into a same plurality of color components, wherein the color components are Y, Cb, and Cr of the YCbCr color space.
(Park (Abstract): “the input RGB image is converted to another color space (e.g., YCbCr) using color space converter (CSC)”. Park teaches the division of the image into a color space, which is the YCbCr color components.
Park (Transitional Network): “a single image can be transformed into multiple representations through color space conversion … we have chosen to use the YCbCr color space for the transitional network. The Y channel denotes the brightness or luminance of the image, while the Cb and Cr channels represent the chrominance.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the division based on color components as taught by Park with the system of Doepke as modified by Veldandi and Henningsson in order allow for different processing strategies for luminance, the structural information, and chrominance, the color information, while keeping the color space in the image processing.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Doepke in view of Veldandi in view of Henningsson and in further view of Shah (US 20160125629 A1).
As per claim 10, Doepke, Veldandi, and Henningsson alone does not explicitly teach the claimed limitations.
However, Doepke, Veldandi, and Henningsson in combination with Shah teaches the claimed:
10. The method according to claim 1, wherein the sets of image data are divided into a plurality of portions based on a user-defined division.
(Shah [0005]: “the interface allowing the user to provide instructions for dividing the electronic image into a plurality of portions by positioning one or more lines dividing the electronic image”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the user instructions as taught by Shah with the system of Doepke as modified by Veldandi and Henningsson in order to allow the user to decide how to divide the image.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Doepke in view of Veldandi in view of Henningsson and in further view of Lee (US 20250029386 A1).
As per claim 11, Doepke, Veldandi, and Henningsson alone do not explicitly teach the claimed limitations.
However, Doepke, Veldandi, and Henningsson in combination with Lee teaches the claimed:
11. The method according to claim 1, wherein the sets of image data are divided into a plurality of regions, the method further comprising:
determining a historic division of the sets of image data in previous sets of image data representing the same image region, and
dividing the sets of image data into a plurality of portions according to the historic division.
(Lee [0018]: “The segmentation system 100 segments objects of each frame using memory of the segmented objects in the previous frames of the video sequence.” Lee teaches the segmentation system that includes a memory that contains previous frames of segmented objects, which correspond to the plurality of portions, and uses the segmented objects of previous frames to divide the current frames in the same way, and since the frames are adjacent, it would also be in the same image region.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the memory of segmented objects as taught by Lee with the system of Doepke as modified by Veldandi and Henningsson in order to be able to reuse previous frames of segmented objects to save and improve computational efficiency.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Doepke in view of Veldandi in view of Henningsson and in further view of 전재열 (KR 101685424 B1), hereinafter Jun.
As per claim 14, Doepke, Veldandi, and Henningsson alone does not explicitly teach the claimed limitations.
However, Doepke, Veldandi, and Henningsson in combination with Jun teaches the claimed:
14. The device according to claim 13, further including a surveillance camera adapted to provide a live stream of panoramic video.
(Jun (page 3, line 26-32): “the panoramic image 133p is formed by combining the unit images 220 corresponding to the field of view (FOV) of the surveillance camera 11. … the panoramic display device 133 is controlled such that only the area of interest is displayed in the panoramic image 133p, View display device 134 is controlled such that only the region of interest is displayed in the live-view image”.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the live stream of a panoramic video as taught by Jun with the system of Doepke as modified by Veldandi and Henningsson in order for the user to view multiple images in a blended panoramic view in real time without needed multiple camera and manual monitoring.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA SUO whose telephone number is (571) 272-8387. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSHUA SUO/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights