Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
This communication is in response to the Application filed on 01/13/2025.
Claims 1-20 are pending.
Claim Objections
Claim 1 and 11 are objected to because of the following informalities:
• In claim 1, the preamble recites, “reconstructing objects without duplicating regions of the objects using transitional images”, but the claim later refers to “an object” in the singular. It would be more consistent to use, “reconstructing an object without duplicating regions of the object using transitional images”
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claim 1, 2, 4, 5, 9, 10, 11, 12, 14, 15, 19 and 20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1, 2, 4, 7, 10, 11, 12, 14, 17, 19 and 20 of U.S. Patent No. 12,198,363 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because:
Claim 1, of instant application and claim 1, U.S. Patent No. 12,198,363 B2 recites common subject matter; Whereby claim 1, of Patent No. 12,198,363 B2 which recites “identifying at least a second portion of the object in a second image and constructing a bounding box around the at least second portion of the object in the second image" does not preclude the elements, “identifying … a second portion of the object in a transitional image and constructing a second bounding box around the second portion of the object in the transitional image” recited by the claim 1 since “the second image” can be broadly interpreted to encompass “the transitional image”, and
Whereby the elements of claim 1, of instant application are fully anticipated by claim 1, of U.S. Patent No. 12,198,363 B2 and anticipation is "the ultimate or epitome of obviousness" (In re Kalm, 154 USPQ 10 (CCPA 1967), also In re Dailey, 178 USPQ 293 (CCPA 1973) and In re Pearson, 181 USPQ 641 (CCPA 1974)).
19/018,708 (instant)
US 12,198,363 B2
Claim 1: A system for reconstructing objects without duplicating regions of the objects using transitional images, comprising:
one or more processors; and one or more computer-readable non-transitory storage media coupled to the one or more processors and comprising instructions that, when executed by the one or more processors cause the system to perform operations comprising:
identifying, via the one or more processors, a first portion of an object in a first image and constructing a first bounding box around the first portion of the object in the first image, wherein the object is a transportation infrastructure component;
identifying, via the one or more processors, a second portion of the object in a transitional image and constructing a second bounding box around the second portion of the object in the transitional image, wherein the first bounding box and the second bounding box are not equivalent;
determining, via an object reconstruction module, equivalent regions within the first image and the transitional image; and
reconstructing the object, via the one or more processors, by excluding the equivalent regions from the object reconstruction
Claim 1: A system for reconstructing objects using transitional images, comprising:
one or more processors; and one or more computer-readable non-transitory storage media coupled to the one or more processors and comprising instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
capturing one or more images of an object, via a sensor, wherein the object is a transportation infrastructure component, a railway environment component, or a railroad track component;
detecting, via an object detection algorithm, the object within the images; identifying at least a first portion of the object in a first image and constructing a bounding box around the at least first portion of the object in the first image;
identifying at least a second portion of the object in a second image and constructing a bounding box around the at least second portion of the object in the second image;
generating, via the one or more processors, transitional images from the images, wherein the transitional images share at least some pixels with at least some of the images and include one or more regions of the object;
determining equivalent regions between the images and one or more transitional images; reconstructing the object, via the one or more processors, from at least some of the images and at least some of the transitional images; and
excluding the equivalent regions from the object reconstruction
Claim 2: The system of Claim 1, wherein the object is a road, railway, airway, waterway, canal, pipeline, terminal component, railroad track, debris, pedestrian, animal, vegetation, joint, switch, frog, rail head, anchor, fastener, gage plate, ballast, or tie.
Claim 2: The system of claim 1, wherein the object is a road, railway, airway, waterway, canal, pipeline, terminal component, railroad track, debris, pedestrian, animal, vegetation, joint, switch, frog, rail head, anchor, fastener, gage plate, ballast, or tie.
Claim 4: The system of Claim 1, wherein the bounding box is an outline of at least a portion of the object.
Claim 4: The system of claim 1, wherein the bounding box is an outline identifying the object.
Claim 5: The system of Claim 1, wherein equivalent regions are regions within two or more images that share the same pixels.
Claim 7: The system of claim 1, wherein the equivalent region is a region within the images and the transitional image that share the same pixels.
Claim 9: The system of Claim 1, wherein detection label equivalency or confidence score thresholds are used to determine that a first region of the object in a first transitional image and a first region of the object in the first image are equivalent regions.
Claim 9: The system of claim 1, wherein detection label equivalency or confidence score thresholds are used to determine that a first region of the object in a first transitional image and a first region of the object in the first image are equivalent regions.
Claim 10: The system of Claim 1, wherein the object reconstruction is a representation of the object in its entirety.
Claim 10: The system of claim 1, wherein the object reconstruction is a representation of the object in its entirety.
Claim 11, corresponds to claim 11 of U. S. Patent No. 12,198,363.
Claim 12, corresponds to claim 12 of U. S. Patent No. 12,198,363.
Claim 14, corresponds to claim 14 of U. S. Patent No. 12,198,363.
Claim 15, corresponds to claim 17 of U. S. Patent No. 12,198,363.
Claim 19, corresponds to claim 19 of U. S. Patent No. 12,198,363.
Claim 20, corresponds to claim 20 of U. S. Patent No. 12,198,363.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “determining, via an object reconstruction module, equivalent regions within the first image and the transitional image” in line 14-15 at claim 1, “the object reconstruction module calculates the IoU of detections in the first image and the transitional image and determines whether the IoU is greater than a predetermined threshold” in line 1-3 at claim 7, “the object reconstruction module determines that the overlap between the first image and the transitional image defines an equivalent region” in line 2-3 at claim 8, “determining, via an object reconstruction module, equivalent regions within the first image and the transitional image” in line 10-11 at claim 11, “the object reconstruction module calculates the IoU of detections in the first image and the transitional image and determines whether the IoU is greater than a predetermined threshold” in line 1-3 at claim 17, and “the object reconstruction module determines that the overlap between the first image and the transitional image defines an equivalent region” in line 2-3 at claim 18.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 and 11 recite, “constructing a first bounding box around a first portion of the object, … constructing a second bounding box around a second portion of the object, … determining, … equivalent regions within the first image and the transitional image.” However, the claim does not clearly define the relationship between the first and second bounding boxes (around the first and second portion of the object) and the equivalent regions. Accordingly, it is unclear how the equivalent regions are identified in relation to the bounding boxes/portions.
Claim 1 and 11 recite, "the object reconstruction" in the last line. There is insufficient antecedent basis for this limitation in the claim. Clarification is required.
Claim 10 and 20 recite, "the object reconstruction" in the 1st line. There is insufficient antecedent basis for this limitation in the claim. Clarification is required.
Claim 1 recites that "the object is a transportation infrastructure component," whereas claim 2 recites that the object is "a road, railway, … debris, … animal." Since "debris" and "animal" are not transportation infrastructure components, claim 2 improperly expands the scope of the claim from which it depends. Consequently, the metes and bounds of the invention cannot be determined with reasonable certainty.
With respect to claim 12, arguments analogous to those presented for claim 2, are applicable.
Claim 6 and 9 recite "detection label equivalency" and "confidence score threshold.” However, the claims and specification do not clearly specify how these criteria are applied to determine whether regions are equivalent, including whether the criteria must be satisfied, and how the confidence score threshold affects the determination. Accordingly, the scope of the claimed determination of equivalent regions cannot be determined with reasonable certainty.
With respect to claim 16, arguments analogous to those presented for claim 6, are applicable.
With respect to claim 19, arguments analogous to those presented for claim 9, are applicable.
Claim 9 recites “a first region of the object” (line 2). It is unclear if “a first region of the object” is referring to “a second portion of the object” (line 10) in claim 1 or something else.
Claim 9 recites “a first transitional image” (line 2). It is unclear if “a first transitional image” is referring to “a transitional image” (line 10-11) in claim 1 or something else.
Claim 9 recites “a first region of the object” (line 3). It is unclear if “a first region of the object” is referring to “a first portion of the object” (line 7) in claim 1 or something else.
Claim 19 recites “a first region of the object” (line 2). It is unclear if “a first region of the object” is referring to “a second portion of the object” (line 6) in claim 11 or something else.
Claim 19 recites “a first transitional image” (line 2). It is unclear if “a first transitional image” is referring to “a transitional image” (line 6-7) in claim 11 or something else.
Claim 19 recites “a first region of the object” (line 3). It is unclear if “a first region of the object” is referring to “a first portion of the object” (line 3) in claim 11 or something else.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 2, 4, 11, 12 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Edpalm et al. (U.S. Publication No. 2018/0338084) (hereafter, "Edpalm") in view of Dick et al. (U.S. Publication No. 2023/0071611) (hereafter, "Dick").
Regarding claim 1, Edpalm teaches a system for reconstructing objects without duplicating regions of the objects using transitional images ([0005] the video camera generating, by joining together portions of the plurality of images, a panorama image corresponding to a union of the fields of view of the plurality of image sensors, wherein, for each overlapping portion of the plurality of images, the video camera includes image data from one of the at least two images that overlap in the panorama image, and the video camera ignores image data from the other images among the at least two images that overlap), comprising: one or more processors ([0051] a processor); and one or more computer-readable non-transitory storage media ([0051] a (non-transitory) computer-readable medium) coupled to the one or more processors and comprising instructions that, when executed by the one or more processors cause the system to perform operations comprising ([0051] In a software implementation, the circuitry may instead be in the form of a processor, such as a microprocessor or a central processing unit, which in association with computer code instructions stored on a (non-transitory) computer-readable medium ... causes the video camera 102 and the client device 104c to carry out any method; [0053] The operation of the system 100): identifying, via the one or more processors, a first portion of an object in a first image and ([0090] if there is an object or a part of an object which covers one of the overlapping portions 502a, 502b, 502c in the original images A, B, C, D; [0057] Step S04 may also comprise identification of non-overlapping portions 504a, 504b, 504c, and 504d) … identifying, via the one or more processors, a second portion of the object in a transitional image and ([0090] if there is an object or a part of an object which covers one of the overlapping portions 502a, 502b, 502c in the original images A, B, C, D; [0057] Step S04 may also comprise identification of non-overlapping portions 504a, 504b, 504c, and 504d) … determining, via an object reconstruction module, equivalent regions within the first image and the transitional image; and ([0056] The images A, B, C, D after transformation and alignment are illustrated as item 500 in FIG. 5. The images A, B, C, D have overlapping portions 502a, 502b, 502c ... In each overlapping portion 502a, 502b, 502c, at least two of the plurality of images A, B, C, D overlap. This means that, for each overlapping portion 502a, 502b, 502c there is image data available from at least two images. In this example, there is image data available from image A and image B in the overlapping portion 502a, from image B and image C in the overlapping portion 502b, and from image C and image D in the overlapping portion 502c. In non-overlapping portions 504a, 504b, 504c and 504d, image data from only one of the images is available, in this case from image A in portion 504a, from image B in the portion 504b, from image C in portion 504c, and from image D in portion 504d; [0055] the panorama image generator 302 may transform and align the plurality of images A, B, C, D shown in FIG. 2, corresponding to the simultaneously received frames of the video streams 308a, 308b, 308c, 308d) reconstructing the object, via the one or more processors, by excluding the equivalent regions from the object reconstruction ([0060] For the overlapping portions 502a, 502b, 502c, image data is available from several of the images A, B, C, D. In this example, image data from two images is available in each overlapping portion ... When generating the portions 512a, 512b, 512c of the panorama image 510, the panorama image generator 302 thus ignores image data from all but the selected image. In this case, image data from image B is ignored when generating portion 512a, image data from image C is ignored when generating portion 512b, and image data from image D is ignored when generating portion 512c. The image data that was ignored by the panorama image generator 302 when generating the panorama image 510; [0019] it may be desirable to only use image data depicting the object from one of the image sensors in order to not have a duplicate of the object in the panorama image).
Edpalm does not expressly teach constructing a first bounding box around the first portion of the object in the first image, wherein the object is a transportation infrastructure component ... constructing a second bounding box around the second portion of the object in the transitional image, wherein the first bounding box and the second bounding box are not equivalent.
However, Dick teaches constructing a first bounding box around the first portion of the object in the first image ([0044] the method 500 optionally includes step 503, which includes identifying one or more regions of interest within each of the images obtained during step 501. For example, referring to FIG. 4, a first region of interest 450 is identified in the image of the railroad. While shown as generally rectangular in FIG. 4, the region(s) of interest can have any suitable size and can be defined by a square boundary, a circular boundary ... the region(s) of interest can be automatically identified within each of the images. For example, if the desired region of interest is a running rail, the system 10 can be configured to automatically identify the running rail within each of the images as the region of interest), wherein the object is a transportation infrastructure component ([0026] The imaging device(s) 16 are generally used to generate image data reproducible as one or more images of the transportation pathway. That is, as the transport device 30 moves along the transportation pathway (e.g., railroad or roadway), the imaging device(s) 16 continuously generate image data) … constructing a second bounding box around the second portion of the object in the transitional image ([0044] the method 500 optionally includes step 503, which includes identifying one or more regions of interest within each of the images obtained during step 501. For example, referring to FIG. 4, a first region of interest 450 is identified in the image of the railroad. While shown as generally rectangular in FIG. 4, the region(s) of interest can have any suitable size and can be defined by a square boundary, a circular boundary ... the region(s) of interest can be automatically identified within each of the images. For example, if the desired region of interest is a running rail, the system 10 can be configured to automatically identify the running rail within each of the images as the region of interest), wherein the first bounding box and the second bounding box are not equivalent ([0044] identifying one or more regions of interest within each of the images obtained … the region(s) of interest can be automatically identified within each of the images. For example, if the desired region of interest is a running rail, the system 10 can be configured to automatically identify the running rail within each of the images as the region of interest; The output bounding boxes will inherently differ due to variations in running rail dimensions and independent spatial coordinates across the multiple images).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Edpalm to incorporate the step/system of identifying regions of interest such as a running rail with square boundaries taught by Dick. In image processing, an identified "region of interest" defined by a geometric boundary such as a "square boundary" structurally and functionally corresponds to a bounding box. When regions in different images vary in local coordinates relative to the detected object portions, they technically meet the description of "not equivalent" bounding boxes.
The suggestion/motivation for doing so would have been to optimize the resource allocation for railway maintenance and repairs ([0061] the user can more efficiently allocate resources for the repair/maintenance of the railroad; [0048] step 504 can include determining distances within each of the plurality of images. In some implementations, in order to improve the accuracy of these distance determinations; [0058] A user viewing graph 601 can easily and efficiently identify locations or stretches of the railroad where there is fouled ballast that requires maintenance (e.g., cleaning or replacement of the ballast). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine Edpalm and Dick to obtain the invention as specified in claim 1.
Regarding claim 2, the combination of Edpalm and Dick teaches all the limitations of claim 1 above. Dick teaches wherein the object is a road, railway, airway, waterway, canal, pipeline, terminal component, railroad track, debris, pedestrian, animal, vegetation, joint, switch, frog, rail head, anchor, fastener, gage plate, ballast, or tie ([0027] the imaging device(s) 16 can capture images of a railroad 200 including a first running rail 210A, a second running rail 210B, a first cross-tie 212A, a second cross-tie 212B, and ballast 214; [0039] The railroad (e.g., subway, elevated train, high speed rail, monorail, tram, etc.) can include one or more running rails, one or more conductor rails, a conductor rail cove, cross-ties (a/k/a ties or sleepers), tie plates, ballast, fasteners, joint bars, welds, switches, overhead power lines, signs (e.g., mileposts, whistle boards, etc.), signals, or any combination thereof).
Regarding claim 4, the combination of Edpalm and Dick teaches all the limitations of claim 1 above. Dick teaches wherein the bounding box is an outline of at least a portion of the object ([0044] the method 500 optionally includes step 503, which includes identifying one or more regions of interest within each of the images obtained during step 501 … referring to FIG. 4, a first region of interest 450 is identified in the image of the railroad. While shown as generally rectangular in FIG. 4, the region(s) of interest can have any suitable size and can be defined by a square boundary, a circular boundary, a triangular boundary, a polygonal boundary, or any combination thereof).
With respect to claim 11, arguments analogous to those presented for claim 1, are applicable.
With respect to claim 12, arguments analogous to those presented for claim 2, are applicable.
With respect to claim 14, arguments analogous to those presented for claim 4, are applicable.
Claim 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Edpalm et al. (U.S. Publication No. 2018/0338084) (hereafter, "Edpalm") in view of Dick et al. (U.S. Publication No. 2023/0071611) (hereafter, "Dick") and further in view of SHIN et al. (U.S. Publication No. 2023/0386043) (hereafter, "SHIN").
Regarding claim 3, the combination of Edpalm and Dick teaches all the limitations of claim 1 above. Dick teaches wherein the first bounding box and the second bounding box are not equivalent ([0044] identifying one or more regions of interest within each of the images obtained … the region(s) of interest can be automatically identified within each of the images. For example, if the desired region of interest is a running rail, the system 10 can be configured to automatically identify the running rail within each of the images as the region of interest; The output bounding boxes will inherently differ due to variations in running rail dimensions and independent spatial coordinates across the multiple images).
The combination of Edpalm and Dick does not expressly teach … because each includes different regions of the object.
However, SHIN teaches the first bounding box and the second bounding box are not equivalent because each includes different regions of the object ([0078] the object detection apparatus may detect an object from each of a plurality of frame images and track the object detected from each of the plurality of frame images, and then may unify identification values of bounding boxes corresponding to a plurality of areas independently tracked for a single object. That is, when the object detection apparatus independently detects and tracks the head and body of the same person, identification values corresponding to the head and body of the same person should be unified).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Edpalm and Dick to incorporate the step/system of detecting a plurality of areas independently tracked for a single object taught by SHIN. Because a person's head and a person's body are fundamentally different regions of the same object (the person), the bounding box for the head and the bounding box for the body are, by definition, "not equivalent" in terms of spatial scope and pixel region.
The suggestion/motivation for doing so would have been to improve the object recognition accuracy ([0004] there is a need for a technique capable of accurately identifying the identity of a person object from a plurality of frame images included in a video). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine Edpalm and Dick with SHIN to obtain the invention as specified in claim 3.
With respect to claim 13, arguments analogous to those presented for claim 3, are applicable.
Claim 5, 9, 15 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Edpalm et al. (U.S. Publication No. 2018/0338084) (hereafter, "Edpalm") in view of Dick et al. (U.S. Publication No. 2023/0071611) (hereafter, "Dick") and further in view of TONG et al. (U.S. Publication No. 2022/0172429) (hereafter, "TONG").
Regarding claim 5, the combination of Edpalm and Dick teaches all the limitations of claim 1 above. Edpalm teaches wherein equivalent regions are regions ([0056] each overlapping portion).
The combination of Edpalm and Dick does not expressly teach … within two or more images that share the same pixels.
However, TONG teaches equivalent regions are regions within two or more images that share the same pixels ([0029] Such object detection provides image regions (e.g., bounding boxes) of the captured image(s) within which an object is detected. The image content of the captured image(s) within the image regions is then compared to the image content of the reconstructed image(s) within the (same) image regions. For example, the comparison may generate a difference metric that measures the difference between the corresponding image regions. Such techniques may be performed for a single image region of a pair of corresponding captured and reconstructed images or across several such image pairs in which the detected object is detected … the term difference metric indicates any image content comparison measure of difference between the image content such as pixel-by-pixel comparison (e.g., sum of absolute differences (SAD), sum of squares of differences, etc.); [0052] pixel-by-pixel comparator 411 may compare image regions 431, 432 in a pixel-by-pixel manner by using SAD, sum of squares of differences, etc. For example, differences in corresponding pixel values between image regions 431, 432 may be used based on RGB values, YUV values, only Y values, etc. to determine a pixel-by-pixel difference metric between image regions 431, 432).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Edpalm and Dick to incorporate the step/system of comparing corresponding regions between each different image using pixel-by-pixel metrics (comparing corresponding pixel values across the same spatial region in different images) taught by TONG.
The suggestion/motivation for doing so would have been to improve the object detection ([0044] Since the cameras of camera array 101 are well synchronized, geometric constraints can be applied to bounding box instances across views for the same detected object for improved bounding box location accuracy and/or object detection in one of 2D images 111 may be used to enhance object detection in another of 2D images 111). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine Edpalm and Dick with TONG to obtain the invention as specified in claim 5.
Regarding claim 9, the combination of Edpalm and Dick teaches all the limitations of claim 1 above. The combination of Edpalm and Dick does not expressly teach wherein detection label equivalency or confidence score thresholds are used to determine that a first region of the object in a first transitional image and a first region of the object in the first image are equivalent regions.
However, TONG teaches wherein detection label equivalency or confidence score thresholds are used to determine that a first region of the object in a first transitional image and a first region of the object in the first image are equivalent regions ([0029] Such object detection provides image regions (e.g., bounding boxes) of the captured image(s) within which an object is detected. The image content of the captured image(s) within the image regions is then compared to the image content of the reconstructed image(s) within the (same) image regions. For example, the comparison may generate a difference metric that measures the difference between the corresponding image regions. Such techniques may be performed for a single image region of a pair of corresponding captured and reconstructed images or across several such image pairs in which the detected object is detected ... the term difference metric indicates any image content comparison measure of difference between the image content such as ... shape comparison metrics (e.g., an indicator of whether the same shape is detected within the image regions and/or confidence values indicating the similarity of such detected shapes), person pose comparison metrics (e.g., an indicator of whether the same body pose is detected within the image regions, confidence values indicating the similarity of such detected poses); [0030] The difference metric for a particular image pair or a combination of difference metrics across pairs of images (e.g., a sum of difference metrics, an average of difference metrics, etc.) is compared to a threshold).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Edpalm and Dick to incorporate the step/system of extracting image regions via bounding boxes, comparing image content using similarity confidence values within those regions between images, and comparing the similarity indicator to a threshold taught by TONG.
Motivation for this combination has been stated in claim 5.
With respect to claim 15, arguments analogous to those presented for claim 5, are applicable.
With respect to claim 19, arguments analogous to those presented for claim 9, are applicable.
Claim 6, 7, 8, 16, 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Edpalm et al. (U.S. Publication No. 2018/0338084) (hereafter, "Edpalm") in view of Dick et al. (U.S. Publication No. 2023/0071611) (hereafter, "Dick") and further in view of Kim et al. (U.S. Publication No. 2023/0119685) (hereafter, "Kim").
Regarding claim 6, the combination of Edpalm and Dick teaches all the limitations of claim 1 above. Edpalm teaches wherein the object reconstruction module ([0055] the panorama image generator 302). The combination of Edpalm and Dick does not expressly teach … uses intersection over union (IoU), detection label equivalency, and/or confidence score thresholds to determine equivalent regions.
However, Kim teaches wherein the object reconstruction module ([0092] the processor 2200 may determine the compression ratio used to generate the reconstructed image as a final compression ratio; [0076] The processor 2200; [0077] The recognition result determination module 2330 may include an overlap degree calculation module 2332, a similarity degree calculation module 2334, and a comparison module 2336) uses intersection over union (IoU), detection label equivalency, and/or confidence score thresholds to determine equivalent regions ([0082] the processor 2200 may calculate the degree of overlap between the first ROI and the second ROI by using the IoU method or the coordinate value error calculation method provided by the overlap degree calculation module 2332).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Edpalm and Dick to incorporate the step/system of using the IoU method to calculate the degree of overlap between different ROIs taught by Kim.
The suggestion/motivation for doing so would have been to improve the object recognition accuracy ([0004] it is desirable to accurately recognize a person's face from a video captured using a camera; [0005] for processing an image or a video captured using a camera, and more particularly, to a device and operating method thereof for determining a compression ratio capable of improving accuracy in recognition of a target object from the image or the video). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine Edpalm and Dick with Kim to obtain the invention as specified in claim 6.
Regarding claim 7, the combination of Edpalm and Dick with Kim teaches all the limitations of claim 6 above. Kim teaches wherein the object reconstruction module ([0092] the processor 2200 may determine the compression ratio used to generate the reconstructed image as a final compression ratio; [0076] The processor 2200; [0077] The recognition result determination module 2330 may include an overlap degree calculation module 2332, a similarity degree calculation module 2334, and a comparison module 2336) calculates the IoU of detections in the first image and the transitional image and ([0079] the overlap degree calculation module 2332 may calculate the degree of overlap among two or more images by using an Intersection Over Union (IoU) method for calculating the degree of overlap using coordinate value information; [0082] the processor 2200 may calculate a degree of overlap between the first ROI including the target object recognized in the raw image and the second ROI including the target object recognized in the reconstructed image, based on coordinate value information of a bounding box of the first ROI and coordinate value information of a bounding box of the second ROI) determines whether the IoU is greater than a predetermined threshold ([0087] The comparison module 2336 is a module configured to compare the degree of overlap between the first and second ROIs, which is calculated using the overlap degree calculation module 2332, with a first threshold, and compare the degree of similarity between the first and second feature vectors, which is calculated using the similarity degree calculation module 2334, with a second threshold. For example, the first threshold may be 0.8, and the second threshold may be 0.5. However, numerical values of the first threshold and the second threshold are not limited to those described above. The processor 2200 may compare the degree of overlap between the first and second ROIs with the first threshold and compare the degree of similarity between the first and second feature vectors with the second threshold by using data and program code in the comparison module 2336).
Regarding claim 8, the combination of Edpalm and Dick with Kim teaches all the limitations of claim 7 above. Kim teaches wherein if the IoU is greater than the predetermined threshold, the object reconstruction module determines that the overlap between the first image and the transitional image defines an equivalent region ([0087] The comparison module 2336 is a module configured to compare the degree of overlap between the first and second ROIs, which is calculated using the overlap degree calculation module 2332, with a first threshold, and compare the degree of similarity between the first and second feature vectors, which is calculated using the similarity degree calculation module 2334, with a second threshold; [0087] The processor 2200 may compare the degree of overlap between the first and second ROIs with the first threshold and compare the degree of similarity between the first and second feature vectors with the second threshold by using data and program code in the comparison module 2336; [0140] when the value of the degree of overlap (R1, R2) exceeds the first threshold and the value of the degree of similarity (f1, f2) exceeds the second threshold, the comparison module 2336 may determine that the target object is correctly recognized in the reconstructed image; [0082] the processor 2200 may calculate a degree of overlap between the first ROI including the target object recognized in the raw image and the second ROI including the target object recognized in the reconstructed image).
With respect to claim 16, arguments analogous to those presented for claim 6, are applicable.
With respect to claim 17, arguments analogous to those presented for claim 7, are applicable.
With respect to claim 18, arguments analogous to those presented for claim 8, are applicable.
Claim 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Edpalm et al. (U.S. Publication No. 2018/0338084) (hereafter, "Edpalm") in view of Dick et al. (U.S. Publication No. 2023/0071611) (hereafter, "Dick") and further in view of Carey et al. (U.S. Publication No. 2021/0350496) (hereafter, "Carey").
Regarding claim 10, the combination of Edpalm and Dick teaches all the limitations of claim 1 above. The combination of Edpalm and Dick does not expressly teach wherein the object reconstruction is a representation of the object in its entirety.
However, Carey teaches wherein the object reconstruction is a representation of the object in its entirety ([0036] Reference is made to FIG. 7, which shows a continuous top-down (x-y plane) view 710 of a conveyor belt 720 constructed from the results of (e.g.) three consecutive image acquisitions in the presence of a long object. The three acquired images 730, 732 and 734 of the overall object are shown in a sequence; [0038] the generation of an actual overall (composite or stitched-together) image from the discrete image acquisitions (snapshots) ... The aggregate feature data can be used independent of creation of an N×M pixel-based overall image to provide appropriate results used to determine object dimensions ... An overall image can be generated and employed in further processes and/or to provide a visual record of all or part of an image object when desired).
It would have been obvious before the effective filing date of the claimed invention to one having ordinary skill in the art to modify the device and method of Edpalm and Dick to incorporate the step/system of generating overall object (overall composite image) from multiple snapshots taught by Carey.
The suggestion/motivation for doing so would have been to improve overall image generation ([0006] With multiple snapshots, a challenge is how to register (stitch together) the plurality of 3D images in an efficient manner so that an overall 3D image accurately represents the object's features; [0007] to capture an accurate measurement of an object larger than a single field of view (FOV) of the sensor; [0048] to generate an accurate set of object dimensions and can operate in the presence of lost or absent 3D data due to complex shapes and/or where multiple objects are present in the conveyor stream). Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predicted results. Therefore, it would have been obvious to combine Edpalm and Dick with Carey to obtain the invention as specified in claim 10.
With respect to claim 20, arguments analogous to those presented for claim 10, are applicable.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL C. CHANG whose telephone number is (571)270-1277. The examiner can normally be reached Monday-Thursday and Alternate Fridays 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan S. Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL C CHANG/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
