DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. KR10-2022-0136678, filed on 10/21/2022.
Information Disclosure Statements
The information disclosure statements (IDS) submitted on 04/18/2025 and 09/02/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Status of Claims
Claims 1-25 filed on 04/18/2025 are presently examined.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“an object recognizer configured to recognize an object” in claims 1 and 16 is described in the specification as [0063] “image processing device 200 may include an image data receiver 210, an object recognizer 220” [0071] “shape recognizers 222, 224 may use the same learning model” [0089] “work apparatus posture detection portion may include an image analysis device (for example, shape recognizer) that analyzes an image of the work apparatus captured through the camera” [FIG. 4].
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
The following patents are discussed below regarding nonstatutory double patenting: U.S. Patent Nos. Lee et al. (US 11541827 B2), Lee et al. (US 12227921 B2), Lee (US 12492533 B2), and Lee et al. (US 12662800 B2). The following copending application is discussed below regarding provisional nonstatutory double patenting: Lee et al. (US 20260152927 A1) App. No. 19/122,334.
In the interest brevity, the mappings and motivations of applied secondary prior art to the above patents and copending application as cited below can be reviewed in the 35 USC § 103 rejections for the corresponding claims. Primary reference Lee et al. (US 20210206330 A1) used in the 35 USC § 103 rejection has a motivation provided when cited in the tables below.
Regarding Lee et al. (US 11541827 B2), hereinafter Lee ‘827, claims 1-2, 9-12, 16-17, and 23-25 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-11 of Lee et al. (US 11541827 B2) hereinafter Lee ‘827 in view of Kim et al. (KR 1566966 B1), hereinafter referred to as Kim. Claims 3-4 and 18-10 are rejected as being unpatentable over Lee ‘827 in view of Kim further in view of Peterson et al. (US 20180134217 A1), hereinafter referred to as Peterson. Claims 5-8 and 20-22 are rejected as being unpatentable over Lee ‘827 in view of Kim further in view of Park et al. (WO 2021054758 A1), hereinafter referred to as Park. Claims 13-15 are rejected as being unpatentable over Lee ‘827 in view of Kim further in view of Lee et al. (US 20210206330 A1), hereinafter Lee ‘6330.
Claims
Patent + Prior Art Mapping
Claims 1 and 16: construction machinery, upper camera in driver cabin, lower camera in rotatably connected front body, forward images from cameras
Lee ‘827 discloses [claims 1, 5, 9]
Claims 1 and 16: recognizing object in both images and determining warning information according to the position
Lee ‘827 does not disclose.
Kim teaches as described in 103 rejection.
Claims 1 and 16: synthesizing the images, transparency processing within a transparency area, displaying the composite image
Lee ‘827 discloses [claims 1-2, 5-6, and 9]
Claims 1 and 16: label the warning information and display the labeled synthesized image
Lee ‘827 does not disclose.
Kim teaches as described in 103 rejection.
Claims 2 and 17: first and second shape recognizers, position determination, warning determiner
Lee ‘827 does not disclose.
Kim teaches as described in 103 rejection.
See obviousness rationale in 35 USC 103.
Claims 3-4 and 18-19: outside/inside labels and differently colored bounding boxes tied to the transparency corresponding regions
Lee ‘827 does not disclose.
Kim + Peterson teach as described in 103 rejection.
Claims 5-8 and 20-22: second image cropping, first and second transparencies, semitransparent second-region synthesis
Lee ‘827 teaches two image synthesis and transparency but does not explicitly disclose claims.
Kim teaches in combination with Peterson as described in 103 rejection.
Claims 9-10 and 23: posture detection and posture dependent transparency, first image transparency when apparatus encroaches a region, and second image transparency with no encroachment
Lee ‘827 discloses [claims 1 and 5]
Claims 11-12 and 24-25: input of image processing condition, switching timing or transparency area, size, relationship to the whole display
Lee ‘827 discloses [claims 3-4, 7-8, and 10-11]
Claims 13-15: captured boom or bucket outline as a line or dotted line, actual captured image transparency processing, selective front body or boom processing
Lee ‘827 fails to disclose in the claims.
Lee ‘6330 (primary reference in 103 rejection) teaches as described in 103 rejection.
Motivation: Lee ‘6330 [0004] “improving forward visibility limited by a work apparatus.”
Regarding Lee et al. (US 12227921 B2), hereinafter Lee ‘921, claims 1-2, 5, 8-17, 20, and 22-25 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 4-16, and 19-25 of Lee ‘217 in view of Kim. Claims 3-4 and 18-19 are rejected as being unpatentable over Lee ‘217 in view of Kim further in view of Peterson. Claims 6-7 and 21 as being unpatentable over Lee ‘217 in view of Kim further in view of Park.
Claims
Patent + Prior Art Disclosure
Claims 1 and 16: construction machinery, upper camera in driver cabin, lower camera in rotatably connected front body, forward images from cameras; synthesizing the images, transparency processing within a transparency area, displaying the composite image
Lee ‘921 discloses [claims 1 and 16]
Claims 1 and 16: recognizing object in both images and determining warning information according to the position; label the warning information and display the labeled synthesized image
Lee ‘921 teaches object recognition [claim 25] but does not disclose warning information/label.
Kim teaches as described in 103 rejection.
Claims 2 and 17: first and second shape recognizers, position determination, warning determiner
Lee ‘921 teaches object recognition in both cameras [claims 1, 11, 16, 25].
Kim teaches as described in 103 rejection.
Claims 3-4 and 18-19: outside/inside labels and differently colored bounding boxes tied to the transparency corresponding regions
Lee ‘921 does not disclose.
Kim + Peterson teach as described in 103 rejection
Claims 5 and 20: crop a second image portion based on the first and synthesize it into the first
Lee ‘921 discloses [claims 13-14 and 22]
Claims 6-7 and 21: corresponding first/second regions, second crop, two transparency values, and same or different transparency values.
Lee ‘921 teaches first region transparency, second crop, and synthesis of regions, but does not disclose specific transparency values for first/second.
Park teaches as described in 103 rejection.
Claims 8 and 22: first/second synthesis regions, semitransparent second region, synthesis into first
Lee ‘921 discloses [claim 15]
Claims 9-12 and 23-25: posture responsive switching and input timing and area conditions
Lee ‘921 discloses [claims 4-7 and 19-21]
Claims 13-15: captured boom or bucket outline as a line or dotted line, actual captured image transparency processing, selective front body or boom processing
Lee ‘921 discloses [claims 8-10]
Regarding Lee (US 12492533 B2), hereinafter Lee ‘533, claims 1-2, 9-17, 23-25 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 3-6, 9, 11-13 of Lee ‘533 in view of Kim. Claims 3-4 and 18-19 are rejected as being unpatentable over Lee ‘533 in view of Kim further in view of Peterson. Claims 5-8 and 20-22 are rejected as being unpatentable over Lee ‘533 in view of Kim further in view of Park.
Claims
Patent + Prior Art Mapping
Claims 1 and 16: construction machinery, upper camera in driver cabin, lower camera in rotatably connected front body, forward images from cameras; synthesizing the images, transparency processing within a transparency area, displaying the composite image
Lee ‘533 discloses [claims 1 and 9]
Claims 1 and 16: recognizing object in both images and determining warning information according to the position; label the warning information and display the labeled synthesized image
Lee ‘533 does not disclose.
Kim teaches as described in 103 rejection.
Claims 2 and 17: first and second shape recognizers, position determination, warning determiner
Lee ‘533 does not disclose.
Kim teaches as described in 103 rejection
Claims 3-4 and 18-19: outside/inside labels and differently colored bounding boxes tied to the transparency corresponding regions
Lee ‘533 does not disclose.
Kim + Peterson teach as described in 103 rejection.
Claims 5-8 and 20-22: second image cropping, first and second transparencies, semitransparent second-region synthesis
Lee ‘533 transparency and synthesis but does not disclose crop / dual regions.
Park teaches as described in 103 rejection.
Claims 9-12 and 23-25: posture responsive transparency based on encroachment of work apparatus, input for switching timing area conditions
Lee ‘533 discloses [claims 3-6, 11-13]
Claims 13-15: outline as line or dotted line of captures boom or bucket outline, selective front body apparatus transparency
Lee ‘533 discloses [claim 1]
Regarding Lee et al. (US 12662800 B2), hereinafter Lee ‘800, claims 1-2, 9-17, and 23 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 3-12 and 14-16 of Lee ‘800 in view of Kim. Claims 3-4, and 18-19 are rejected as being unpatentable over Lee ‘800 in view of Kim further in view of Peterson. Claims 5-8, 20-22 are rejected as being unpatentable over Lee ‘800 in view of Kim further in view of Park. Claims 24-25 are rejected as being unpatentable over Lee ‘800 in view of Kim further in view of Lee ‘6330.
Claims
Patent + Prior Art Mapping
Claims 1 and 16: construction machinery, upper camera in driver cabin, lower camera in rotatably connected front body, forward images from cameras; synthesizing the images, transparency processing within a transparency area, displaying the composite image
Lee ‘800 discloses [claims 1 and 12]
Claims 1 and 16: recognizing object in both images and determining warning information according to the position; label the warning information and display the labeled synthesized image
Lee ‘800 teaches object recognition but does not disclose position based warning/labeling.
Kim teaches as described in 103 rejection.
Claims 2 and 17: first and second shape recognizers, position determination, warning determiner
Lee ‘800 does not disclose.
Kim teaches as described in 103 rejection.
Claims 3-4 and 18-19: outside/inside labels and differently colored bounding boxes tied to the transparency corresponding regions
Lee ‘800 does not disclose.
Kim + Peterson teach as described in 103 rejection.
Claims 5-8 and 20-22: second image cropping, first and second transparencies, semitransparent second-region synthesis
Lee ‘800 does not disclose.
Park teaches as described in 103 rejection.
Claims 9-12 posture-responsive transparency and dependency of the encroachment of the work apparatus within an area, input affecting switching timing and area
Lee ‘800 discloses [claims 3-6]
Claims 13-15: outline as line or dotted line of captures boom or bucket outline, selective front body apparatus transparency
Lee ‘800 discloses [claims 7-9]
Claim 23: detect apparatus posture and transparency process based on detection
Lee ‘800 discloses [claim 14]
Claims 24-25 set condition to make first and second images transparent, condition including switching timing or transparency area
Lee ‘800 does not disclose.
Lee ‘6330 (primary reference in 103 rejection) teaches as described in 103 rejection.
Motivation: Lee ‘6330 [0004] “improving forward visibility limited by a work apparatus.”
Claim 1-2, 5-17, and 20-25 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 4-13, and 16-20 of copending Application No. Lee et al. (19/122,334), hereinafter referred to as Lee ‘334, in view of Kim. Claims 3-4 and 18-19 are provisionally rejected as being unpatentable over Lee ‘334 in view of Kim, further in view of Peterson. This is a provisional nonstatutory double patenting rejection.
Claims
Patent + Prior Art Mapping
Claims 1 and 16: construction machinery, upper camera in driver cabin, lower camera in rotatably connected front body, forward images from cameras; synthesizing the images, transparency processing within a transparency area, displaying the composite image
Lee ‘334 discloses [claim 1 and 13]
Claims 1 and 16: recognizing object in both images and determining warning information according to the position; label the warning information and display the labeled synthesized image
Lee ‘334 does not disclose.
Kim teaches as described in 103 rejection.
Claims 2 and 17: first and second shape recognizers, position determination, warning determiner
Lee ‘334 does not disclose.
Kim teaches as described in 103 rejection.
Claims 3-4 and 18-19: outside/inside labels and differently colored bounding boxes tied to the transparency corresponding regions
Lee ‘334 does not disclose.
Kim + Peterson teach as described in 103 rejection.
Claims 5 and 20: crop a second image portion based on the first and synthesize it into the first
Lee ‘334 discloses [claim 4 and 16]
Claims 6-8 and 21-22: corresponding first/second regions, second crop, two transparency values, and same or different transparency values; first/second synthesis regions, semitransparent second region, synthesis into first
Lee ‘334 discloses [claims 5-7, 13, and 17]
Claims 9-12 and 23-25: posture responsive transparency based on encroachment of work apparatus, input for switching timing area conditions
Lee ‘334 discloses [claims 8-11 and 18-20]
Claims 13-15: outline as line or dotted line of captures boom or bucket outline, selective front body apparatus transparency
Lee ‘334 discloses [claim 9 and 12]
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 9-17, and 23-25 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US 20210206330 A1) in view of Kim et al. (KR 1566966 B1), hereinafter referred to as Lee and Kim, respectively.
Regarding claims 1 and 16, Lee discloses A control system for construction machinery, the control system comprising: ([claim 1] “A control system for construction machinery, the control system comprising:”)
an upper camera installed in a driver cabin of a rear body to capture a front view of the driver cabin ([claim 1] “an upper camera installed in a driver cabin to photograph the front of the driver cabin”);
a lower camera installed in a front body rotatably connected to the rear body to capture a front view of the front body ([claim 1] “a lower camera installed in a vehicle body to photograph the front of the vehicle body” [0034] “a front body 12 and a rear body 14 rotatably connected to each other.”);
an image processor configured to synthesize the first and second images into one synthesized image ([claim 1] “an image processing portion configured to synthesize first and second images captured from the upper camera and the lower camera into one image”), and perform transparency processing on at least one of the first and second images ([claim 1] “transparency-process at least one of the first and second images in the synthesized image”) within a transparency processing area of the synthesized image ([0054] “In the transparency processing, the portions of the first image and/or the second image within the transparency processing area”).
Lee fails to explicitly disclose an object recognizer configured to recognize an object in first and second images captured by the upper camera and the lower camera, respectively, and to determine warning information according to a position of the recognized object and labeling processing on the warning information of the object; and a display device configured to display the synthesized image labeled with the warning information.
However, Kim teaches an object recognizer configured to recognize an object in first and second images captured by the plurality of cameras ([0047] “In FIG. 3, multiple cameras (110a, 110b, 110c, and 110d) are positioned” [0034] “Multiple cameras (110) can be moved up and down” [0051] “The pedestrian recognition unit (130) detects moving objects based on a series of work environment views (step S203).”), and to determine warning information according to a position of the recognized object and labeling processing on the warning information of the object; and a display device configured to display the synthesized image labeled with the warning information ([0050] “The work environment view providing unit (120) can display markers for the moving object in a series of work environment views depending on whether the moving object is within the work safety guidelines if the moving object corresponds to a pedestrian.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee with Kim’s teaching of a similar work machine with a plurality of cameras which detects moving objects within a work environment and recognizes pedestrians and uses warning markers to visually display a warning based on safety guidelines. Lee seeks to reduce blind spots and prevent accidents ([0086] “remove a blind spot that is obscured by the front work apparatus. Thus, the operator's cognitive ability may be increased to secure stability, to thereby prevent safety accidents.”). One would be motivated, with reasonable expectation of success, to use Lee’s upper and lower synthesized camera images and Kim’s recognition system to recognize pedestrians and provide a visual marker warning the work machine operator of nearby pedestrians in order to prevent accidents and injury by providing greater awareness to the machine operator via the display (Kim [0002] “heavy equipment has a high driver's seat, making it difficult for the operator to accurately perceive surrounding objects. Accordingly, heavy equipment is equipped with cameras that film the surrounding area, enabling the operator to perceive surrounding objects.”).
Regarding claims 2 and 17, Lee discloses The control system for construction machinery of claim 1, wherein there is a first and second camera capturing a first and second image ([claim 1] “an image processing portion configured to synthesize first and second images captured from the upper camera and the lower camera into one image”).
Lee fails to disclose the object recognizer includes:
a first shape recognizer configured to recognize the object in the first image and determine the position of the object; a second shape recognizer configured to recognize the object in the second image and determine the position of the object; and a warning information determiner configured to determine warning information of the recognized object based on the position of the object.
However, Kim teaches a shape recognizer configured to recognize the object in the first image and determine the position of the object ([0051] “In FIG. 6, when a specific object (610) is recognized in a series of work environment views, the pedestrian recognition unit (130) can detect an edge (620) for the specific object in an image included in the series of work environment views and associated with the specific object” [0039] “Edges are parts where brightness changes abruptly in a series of working environment views, which can make it possible to know the shape, size, and location of at least one object.”);
a warning information determiner configured to determine warning information of the recognized object based on the position of the object ([0038] “display a marker for the moving object as danger in a series of work environment views if the moving object is within the work safety guidelines” [0050] “FIG. 5 … displaying a marker for the moving object in red on a series of work environment views if the moving object is within the work safety guidelines”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee with Kim’s teaching of a similar work machine with a plurality of cameras which detects moving objects within a work environment and recognizes pedestrians and uses warning markers to visually display a warning based on safety guidelines. Lee seeks to reduce blind spots and prevent accidents ([0086] “remove a blind spot that is obscured by the front work apparatus. Thus, the operator's cognitive ability may be increased to secure stability, to thereby prevent safety accidents.”). One would be motivated, with reasonable expectation of success, to use Lee’s upper and lower synthesized camera images and Kim’s recognition system to recognize pedestrians and provide a visual marker warning the work machine operator of nearby pedestrians in order to prevent accidents and injury by providing greater awareness to the machine operator via the display (Kim [0002] “heavy equipment has a high driver's seat, making it difficult for the operator to accurately perceive surrounding objects. Accordingly, heavy equipment is equipped with cameras that film the surrounding area, enabling the operator to perceive surrounding objects.”).
In regard to a second shape recognizer configured to recognize the object in the second image and determine the position of the object, while Kim does not explicitly teach two separate shape recognizers, being a first and second recognizer, this is predictable duplication of parts. Kim’s object recognizer is used on the plurality of cameras and their corresponding images (Kim [at least 0051]). Lee teaches two cameras: the upper and lower camera. Applying the object recognizer taught by Kim to the two cameras and their corresponding images taught by Lee would be obvious and result in a predictable outcome: an object being recognized in both corresponding camera feeds. See MPEP 2144.04(VI)(B) for more details.
Regarding claims 9 and 23, Lee discloses The control system for construction machinery of claim 1, further comprising
a work apparatus posture detection portion configured to detect a posture of a front work apparatus ([0049] “the work apparatus posture detection portion may include a boom angle sensor 24 for detecting the position of the bucket 30 or the posture of the boom 20 … the work apparatus posture detection portion may include an image analysis device that analyzes an image of the work apparatus captured through the camera to determine the posture of the work device.”),
wherein the image processor is configured to transparency-process at least one of the first and second images to be transparent in the transparency processing area according to the posture of the front work apparatus detected by the work apparatus posture detection portion ([0052] “The transparency processor 220 may perform transparency processing on at least one of the first and second images in the synthesized image according to the detected posture of the work apparatus.”).
Regarding claim 10, Lee discloses The control system for construction machinery of claim 9, wherein the image processor is configured to transparency-process the first image to be transparent in the transparency processing area when at least a portion of the front working apparatus encroaches on a predetermined position, and to transparency-process the second image transparent in the transparency processing area when the front working apparatus does not encroach on the predetermined position ([0055] “When the bucket or boom position is lower than a predetermined position (transparency switching position), which can be determined that the at least a part of the work apparatus does not invade the transparency processing area, the second image in the synthesized image may be transparency-processed to be transparent. On the other hand, when the bucket or boom position is the predetermined position (transparency switching position), which can be determined that the at least a part of the work apparatus invades the transparency processing area, the first image in the synthesized image may be transparency-processed to be transparent.”).
Regarding claims 11 and 24, Lee discloses The control system for construction machinery of claim 1, further comprising an input portion configured to set an image processing condition in the image processor ([0060] “an image processing condition in the image processing portion 200 may be set through an input portion 400.” [0060] “As the transparency processing area is determined, the transparency switching position of the first and second images, the transparency processing area in the entire display area of the display device 300, and the like may be set.”).
Regarding claims 12 and 25, Lee discloses The control system for construction machinery of claim 11, wherein the image processing condition includes a transparency processing switching timing of the first and second images or an area occupied by the transparency processing area within the entire display area of the display device ([0060] “the transparency switching position of the first and second images, the transparency processing area in the entire display area of the display device 300 … the transparency switching timing, etc. may be fixedly set by a manufacturer according to a type of equipment, and may be freely changed and set by the operator or maintenance personnel.”).
Regarding claim 13, Lee discloses The control system for construction machinery of claim 1, wherein the image processor is configured to process an outline of an appearance of a boom or bucket that is captured and transparency-processed in the first image or the second image in the transparency processing area to be displayed as a line or a dotted line ([FIG 5] When below zone A, the bucket is a solid line outline. When inside zone A, the bucket is dotted line. [0074] “checking whether a part of the bucket 30 or the boom 20 is located within the transparency processing area A” [0079] “the first image captured from the upper camera 110 may be transparency-processed, so that an object implemented by the lower camera 120 may be displayed as a main point (focus) within the transparency processing area A”).
Regarding claim 14, Lee discloses The control system for construction machinery of claim 13, wherein the outline of the transparency-processed boom or bucket is displayed by actually transparency-processing a boom or bucket image captured in the first image or the second image ([FIG 6] seeing that the object is behind the bucket, the bucket is transparency processed to enable the driver to see the object – the pedestrian. [0074] “checking whether a part of the bucket 30 or the boom 20 is located within the transparency processing area A” [0079] “the first image captured from the upper camera 110 may be transparency-processed, so that an object implemented by the lower camera 120 may be displayed as a main point (focus) within the transparency processing area A … When the bucket 30 or the boom 20 is in a relatively high position, a portion of the front work apparatus obscuring the front view in the first image may be transparency-processed so that the object O may be identified in the synthesized image.”).
Regarding claim 15, Lee discloses The control system for construction machinery of claim 1, wherein the image processor is configured to selectively transparency-process a boom or bucket image coupled to the front body among the first and second images in the transparency processing area ([0074] “checking whether a part of the bucket 30 or the boom 20 is located within the transparency processing area A” [0079] “the first image captured from the upper camera 110 may be transparency-processed, so that an object implemented by the lower camera 120 may be displayed as a main point (focus) within the transparency processing area A … When the bucket 30 or the boom 20 is in a relatively high position, a portion of the front work apparatus obscuring the front view in the first image may be transparency-processed so that the object O may be identified in the synthesized image.”).
Claims 3-4 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Kim as applied to claims 1 and 16 above, further in view of Peterson et al. (US 20180134217 A1), hereinafter referred to as Peterson.
Regarding claims 3 and 18, Lee discloses The control system for construction machinery of claim 2, wherein the image processing matches the first and second images of the upper and lower cameras and determines the transparency-area bounding box, displaying the first image outside that area and the second image inside that area ([0052] “match the first image and the second image captured by the upper camera 110 and the lower camera 120 to find portions of images that overlap (are duplicated) in the first and second images and synthesize the overlapping portions of the images into one synthesized image.” [0060] “when the bucket 30 moves to be located at the boundary of the transparency processing area, the bucket 30 may be considered to be located at a predetermined position for transparency switching.”) and divides the display into transparency and external areas, displaying second-image content as a focus region ([0062] “dividing the image captured by the camera portion into the transparency processing area A and an external area of the transparency processing area A.” [0063] “the first image in the external area of the transparency processing area A, and may display a transparency image in which at least one of the first image and the second image is displayed as a focus”).
Lee fails to disclose determine the warning information as a first label when the recognized object is positioned outside a first region in the first image corresponding to the transparency processing area, and determine the warning information as a second label when the recognized object is positioned within a second region in the second image corresponding to the transparency processing area.
However, Peterson teaches determine the warning information as a first label when the recognized object is positioned outside a first region in the first image corresponding to the transparency processing area ([0017] “FIG. 5 is another view of the display screen of FIG. 4, showing an outline or box to indicate when an object is not in the blind spot region” [0067] “a main view to the rear of the vehicle”), and
determine the warning information as a second label when the recognized object is positioned within a second region in the second image corresponding to the transparency processing area ([0067] “In addition to the main view, a smaller view in the upper outboard corner shows an area typically covered by a ‘spotter’ or blind-zone mirror.” [0068] “the static spotter function may include a red highlight marker or box (FIG. 4) that indicates when a car is in the blind spot.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee with Peterson’s teaching of using two different warning labels depending on where the detected vehicle is, whether inside or outside a smaller blind spot area of a larger main view. Lee seeks to reduce blind spots and prevent accidents ([0086] “remove a blind spot that is obscured by the front work apparatus. Thus, the operator's cognitive ability may be increased to secure stability, to thereby prevent safety accidents.”). One would be motivated, with reasonable expectation of success, to provide these warning labels depending on where the detected vehicle is in the image frame in order to improve awareness for the driver (Peterson [0005] “The displayed graphic overlay thus provides enhanced cognitive awareness for the driver” [0157] “changing the color of or brightness of the detected object or vehicle, to enhance the driver's cognitive awareness of the detected object or vehicle”). This would help to prevent accidents with objects in blind spots of the operator’s field of view inside the work machine.
Regarding claim 4 and 19, Lee fails to disclose The control system for construction machinery of claim 3, wherein the image processor is configured to: label the recognized object with a bounding box of a first color when the recognized object is positioned outside the first region in the first image, and label the recognized object with a bounding box of a second color different from the first color when the recognized object is positioned within the second region in the second image.
However, Peterson teaches label the recognized object with a bounding box of a first color when the recognized object is positioned outside the first region in the first image, and label the recognized object with a bounding box of a second color different from the first color when the recognized object is positioned within the second region in the second image ([0076] “The overlay can be made up of one or more colors to indicate different distances and safe or unsafe zones” [0089] “In the illustrated embodiment of FIG. 17B, the highlighting is a colored box around the detected vehicle” [0157] “when an approaching vehicle enters the blindzone area … alert the driver … by changing the color of or brightness of the detected object”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee with Peterson’s teaching of using different colored warning label bounding boxes depending on where the detected vehicle is, whether inside or outside a smaller blind spot area of a larger main view. Lee seeks to reduce blind spots and prevent accidents ([0086] “remove a blind spot that is obscured by the front work apparatus. Thus, the operator's cognitive ability may be increased to secure stability, to thereby prevent safety accidents.”). One would be motivated, with reasonable expectation of success, to provide these differently colored warning label bounding boxes depending on where the detected vehicle is in the image frame in order to improve awareness for the driver (Peterson [0005] “The displayed graphic overlay thus provides enhanced cognitive awareness for the driver” [0157] “changing the color of or brightness of the detected object or vehicle, to enhance the driver's cognitive awareness of the detected object or vehicle”). This would help to prevent accidents with objects in blind spots of the operator’s field of view inside the work machine.
Claims 5-8 and 20-22 are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Kim as applied to claims 1 and 16 above, further in view of Park et al. (WO 2021054758 A1), hereinafter referred to as Park.
Regarding claims 5 and 20, Lee fails to disclose The control system for construction machinery of claim 1, wherein the image processor is configured to crop a portion of the second image based on the first image and to synthesize the cropped image of the second image into the first image.
However, Park teaches wherein the image processor is configured to crop a portion of the second image based on the first image ([0031] “The image processor (150) sets a composite area (CA1) in the center of the first front image (FI1).” [0032] “image processor (150) detects an image of a composite target area (CA2) corresponding to a composite area (CA1) of a first front image (FI1) from a second front image (FI2).”) and to synthesize the cropped image of the second image into the first image ([0032] “The image processor (150) synthesizes the image of the synthesis target area (CA2) of the second front image (FI2) into the synthesis area (CA1) of the first front image (FI1) to generate a synthesized front image (CFI)”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee with Park’s teaching of two cameras on a work machine used for the detection of objects in blind spots by using a cropped image of a second front image based on the first image of a first camera. One would be motivated, with reasonable expectation of success, to synthesize the two images into one using a cropped second image based on the first image in order to indicate the exact location of an object in a blind spot of the first camera that is obstructed by the work equipment, and also to reduce computation requirement by using a cropped image of the second camera (Park [0033] “the image of the composite target area (CA2) is a blind spot caused by the bucket (12), and since the object in the image of the composite target area (CA2) is an object located in the blind spot, it is important to indicate the exact location.” [0037] “the image processor (150) composites a part of the second front image (FI2) (i.e., the image of the composite target area (CA2)) to the central part (i.e., the composite area (CA1)) of the first front image (FI1), which is the area where the visual obstruction component is mainly located, thereby minimizing the amount of computation for image processing and enabling fast image processing”).
Regarding claims 6 and 21, Lee fails to disclose The control system for construction machinery of claim 1, wherein the image processor is configured to:
transparency-process a first region in the first image corresponding to the transparency processing area to have a first transparency;
determine a second region in the second image corresponding to the transparency processing area as a crop region, and transparency-process the determined second region to have a second transparency; and
synthesize the cropped image extracted from the second image into the first image.
However, Park teaches transparency-process a first region in the first image corresponding to the transparency processing area to have a first transparency ([0031] “The image processor (150) sets a composite area (CA1) in the center of the first front image (FI1).” [0065] “the image processor (150) sets the image transparency of the second front image (FI2) to approximately 40% and the image transparency of the first front image (FI1) to approximately 60% to generate a composite front image (CFI), as an example”);
determine a second region in the second image corresponding to the transparency processing area as a crop region, and transparency-process the determined second region to have a second transparency ([0032] “The image processor (150) detects an image of a composite target area (CA2) corresponding to a composite area (CA1) of a first front image (FI1) from a second front image (FI2).” [0065] “the image processor (150) sets the image transparency of the second front image (FI2) to approximately 40% and the image transparency of the first front image (FI1) to approximately 60% to generate a composite front image (CFI), as an example”); and
synthesize the cropped image extracted from the second image into the first image ([0032] “The image processor (150) synthesizes the image of the synthesis target area (CA2) of the second front image (FI2) into the synthesis area (CA1) of the first front image (FI1) to generate a synthesized front image (CFI).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee with Park’s teaching of two cameras on a work machine used for the detection of objects in blind spots by using a cropped image of a second front image based on the first image of a first camera and applying different transparencies to the views. One would be motivated, with reasonable expectation of success, to synthesize the two images into one using a cropped second image based on the first image with different transparencies in order to indicate the exact location of an object in a blind spot of the first camera that is obstructed by the work equipment, which would result in predictable outcome of indicating that the object is present in the synthesis target area with the lower transparency causing the synthesis target area appearing clearer (Park [0033] “the image of the composite target area (CA2) is a blind spot caused by the bucket (12), and since the object in the image of the composite target area (CA2) is an object located in the blind spot, it is important to indicate the exact location.” [0065] “since it is important to indicate whether an object is present in the image in the synthesis target area (CA2), the image processor (150) sets the image transparency of the synthesis target area (CA2) to a relatively low level to generate a composite front image (CFI) in which the image of the synthesis target area (CA2) appears clearer than the image of the synthesis area (CA1).”).
Regarding claim 7, Lee fails to explicitly disclose discloses The control system for construction machinery of claim 6, wherein the first transparency and the second transparency are set to be the same or different from each other.
However, Park teaches the first transparency and the second transparency are set to be the same or different from each other ([0064] “the image processor (150) sets the image transparency of the synthesis area (CA1) and the synthesis target area (CA2) to 50% each to generate a composite front image (CFI), as an example.” [0065] “the image processor (150) sets the image transparency of the second front image (FI2) to approximately 40% and the image transparency of the first front image (FI1) to approximately 60% to generate a composite front image (CFI), as an example”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee with Park’s teaching of two cameras on a work machine used for the detection of objects in blind spots by using a cropped image of a second front image based on the first image of a first camera and applying different transparencies to the views. One would be motivated, with reasonable expectation of success, to synthesize the two images into one using a cropped second image based on the first image with different transparencies in order to indicate the exact location of an object in a blind spot of the first camera that is obstructed by the work equipment, which would result in predictable outcome of indicating that the object is present in the synthesis target area with the lower transparency causing the synthesis target area appearing clearer (Park [0033] “the image of the composite target area (CA2) is a blind spot caused by the bucket (12), and since the object in the image of the composite target area (CA2) is an object located in the blind spot, it is important to indicate the exact location.” [0065] “since it is important to indicate whether an object is present in the image in the synthesis target area (CA2), the image processor (150) sets the image transparency of the synthesis target area (CA2) to a relatively low level to generate a composite front image (CFI) in which the image of the synthesis target area (CA2) appears clearer than the image of the synthesis area (CA1).”).
Regarding claims 8 and 22, Lee fails to disclose The control system for construction machinery of claim 1, wherein the image processor is configured to
set a portion of the first image as a first synthesizing region, set a portion of the second image as a second synthesizing region,
process the second synthesizing region of the second image to be semi-transparent, and synthesize the semi-transparent second synthesizing region into the first synthesizing region of the first image to create the synthesized image.
However, Park teaches set a portion of the first image as a first synthesizing region, set a portion of the second image as a second synthesizing region ([0031] “The image processor (150) sets a composite area (CA1) in the center of the first front image (FI1).” [0032] “image processor (150) detects an image of a composite target area (CA2) corresponding to a composite area (CA1) of a first front image (FI1) from a second front image (FI2).”),
process the second synthesizing region of the second image to be semi-transparent, and synthesize the semi-transparent second synthesizing region into the first synthesizing region of the first image to create the synthesized image ([0065] “The image processor (150) may generate a composite front image (CFI) by setting the image transparency of the composite area (CA1) of the first front image (FI1) higher than the image transparency of the composite target area (CA2) of the second front image (FI2).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee with Park’s teaching of two cameras on a work machine used for the detection of objects in blind spots by using a cropped image of a second front image based on the first image of a first camera. One would be motivated, with reasonable expectation of success, to synthesize the two images into one using a cropped second image based on the first image in order to indicate the exact location of an object in a blind spot of the first camera that is obstructed by the work equipment, and also to reduce computation requirement by using a cropped image of the second camera (Park [0033] “the image of the composite target area (CA2) is a blind spot caused by the bucket (12), and since the object in the image of the composite target area (CA2) is an object located in the blind spot, it is important to indicate the exact location.” [0037] “the image processor (150) composites a part of the second front image (FI2) (i.e., the image of the composite target area (CA2)) to the central part (i.e., the composite area (CA1)) of the first front image (FI1), which is the area where the visual obstruction component is mainly located, thereby minimizing the amount of computation for image processing and enabling fast image processing”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARK R HEIM whose telephone number is (571)270-0120. The examiner can normally be reached M-F 9-6 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fadey Jabr can be reached on 571-272-1516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.R.H./Examiner, Art Unit 3668
/Fadey S. Jabr/Supervisory Patent Examiner, Art Unit 3668
Prosecution Insights