DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in the Republic of Korea on 08/09/2023. It is noted, however, that applicant has not filed a certified copy of the KR10-2023-0114001 as required by 37 CFR 1.55.
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in the Republic of Korea on 12/12/2023. It is noted, however, that applicant has not filed a certified copy of the KR10-2023-0179969 as required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 09/04/2024, 02/10/2025 and 11/06/2025 was filed before the first Office action on the merits. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: output device to output the optimal door bin image in at least claim 6.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
output device 120 may include a display 123, Applicant Specification 0099
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 6, 7, 9-12, 16 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (US20230043011A1) in view of Takeshi et al. (WO2022038799A1).
Regarding Claim 1, Kim teaches a refrigerator [refrigerator 10, Figure 5A] comprising:
a body [cabinet 11, Figure 5A] comprising a storage chamber [refrigerating compartment 18, Figure 5A];
a door [refrigerator door 12, Figure 5A] rotatably coupled to the body to open or close the storage chamber [where door 12 is rotatably provided in front of the refrigerating compartment, Figure 5A;0113], the door comprising a door bin [where a storage space for accommodating foods may be in each of the inner door 13 and the outer door 14 of refrigerator door 12, Figure 5A; 0281];
a camera on the body [camera 121 provided on a ceiling of cabinet 11, Figure 5B; 0120], the camera being configured to capture an image of an inside of the body [where the camera may photograph the inside of refrigerator 10;0120];
a memory [memory 230, Figure 2] configured to store one or more instructions [where one or more instructions may be stored in memory 230 of AI server 200; 0079]; and
at least one processor operatively coupled to the memory [processor 260 of AI server 200 and processor 180 of AI device 100, Figure 2 and Figure 4] and configured to execute the one or more instructions stored in the memory [where one or more instructions constitute the learning model stored in memory 230; where AI device 100 may infer the result value for the input data using the learning model and generate the response or control command based on the result; 0079;0080], wherein the one or more instructions, when executed by the at least one processor [where AI server 200 may transmit the learning model to AI device 100 which includes processor 180, Figure 2 and Figure 4; 0087], cause the refrigerator to:
based on detecting opening of the storage, control the camera to capture the image of at least a part of the inside of the body [where when opening of the storage rooms 510 and 530 is detected the processor 180 transmits a command for operation of camera 121 which may acquire an image of the first storage room 510 or second storage room 530;0169;0170], identify, based on the image, whether at least one of a first area of the inside of the body included in the image satisfies a predetermined condition [where processor 180 extracts an image having the largest surface area of each of the storage rooms; 0182; 0183], and based on identifying that the predetermined condition is satisfied, store the image as an optimal door bin image [where the processed image frame may be stored in the memory 170;0096].
Kim does not explicitly teach the camera is configured to capture an image of an inside of the door where the one or more instructions cause the refrigerator to, based on detecting opening of the door, control the camera to capture the image of the door.
However, Takeshi teaches a method of processing image data in a refrigerator [0001] including where the camera [image pickup device 30, Figure 1;0030] is configured to capture an image of an inside of the door [door storage section 11a and 11b, Figure 11;0038] where the one or more instructions [where control SoC 41 issues a shooting instruction to camera 33; 0023] cause the refrigerator to, based on detecting opening of the door [when SoC41 detects the door is opened; 0023], control the camera to capture the image of the door [0023] where one of ordinary skill in the art would have been capable of applying this known technique, a wide-angle fisheye lens and ensuring the door is open, to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art i.e., allowing the entire interior of the fridge to be viewed [Takeshi; 0018].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the assembly of Kim to have where the camera is configured to capture an image of an inside of the door where the one or more instructions cause the refrigerator to, based on detecting opening of the door, control the camera to capture the image of the door in view of the teachings of Takeshi where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable i.e., allowing the entire interior of the fridge to be viewed [Takeshi; 0018].
Regarding Claim 2, Kim, as modified, teaches the invention of claim 1 and further teaches wherein the camera [camera 121, Figure 5B] is on an upper end of the body [where camera 121 is provided on a ceiling of the cabinet 11 to photograph a lower side; 0120], wherein the first area of the inside of the body is an area on a lower end [where camera 121 photographs a lower side, Figure 5B; 0120] but Kim does not explicitly teach the first area includes a side surface of the body covered by the door in a state in which the door is closed and a first line, included in the door included in the image as recited in an alternative in claim 1, wherein the first line is on an upper end of the door.
However, Takeshi teaches a method of processing image data in a refrigerator [0001] including where the camera [image pickup device 30, Figure 1;0030] is configured to capture the first area [storage compartment 11c, Figure 11; 0038] including a side surface of the body covered by the door in a state in which the door is closed [a side of refrigerator 100 with box body 10, Figure 1 and annotated Figure 11] and a first line included in the door included in the image wherein the first line is on an upper end of the door [a first line of door storage section 11b, annotated Figure 11, where a door inherently includes lines to form its geometry] one of ordinary skill in the art would have been capable of applying this known technique, a wide-angle fisheye lens, to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art i.e., allowing the entire interior of the fridge to be viewed with a single camera [Takeshi; 0018].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the assembly of Kim to have the first area includes a side surface of the body covered by the door in a state in which the door is closed and wherein the first line is on an upper end of the door in view of the teachings of Takeshi where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable i.e., allowing the entire interior of the fridge to be viewed with a single camera [Takeshi; 0018].
Regarding Claim 6, Kim, as modified, teaches the invention of claim 1 and further teaches an output device [output unit 150; 0062], wherein the one or more instructions [where one or more instructions constitute the learning model stored in memory 230; where AI device 100 may infer the result value for the input data using the learning model and generate the response or control command based on the result; 0079;0080], when executed by the at least one processor [processor 260 of AI server 200 and processor 180 of AI device 100, Figure 2 and Figure 4], further cause the refrigerator to:
based on receiving a user input [via user input unit 123; 0099] to output the optimal door bin image [0305], control the output device to output the optimal door bin image [where the image display area 1912 may be an area for displaying images corresponding to areas selected by the user among storage identification areas; 0305-0308].
Regarding Claim 7, Kim, as modified, teaches the invention of claim 6 and further teaches where the door [refrigeration compartment door 12, Figure 5A] further comprises a first door and a second door [where the refrigerator compartment door 12 is provided as a pair and may be rotatably connected to the left and right edges of the front surface of the cabinet 11, respectively, Figure 5A; 0115], and wherein the one or more instructions [where one or more instructions constitute the learning model stored in memory 230; where AI device 100 may infer the result value for the input data using the learning model and generate the response or control command based on the result; 0079;0080], when executed by the at least one processor [processor 260 of AI server 200 and processor 180 of AI device 100, Figure 2 and Figure 4], further cause the refrigerator to:
based on detecting the storage is opened [where when opening of the storage rooms 510 and 530;0169;0170], control the camera to obtain an optimal door bin image corresponding to the one of the first door or the second door [where the processor 180 transmits a command for operation of camera 121 which may acquire an image of the first storage room 510 or second storage room 530;0169;0170], and control the output device to output the optimal door bin image corresponding to the one of the first door or the second door [where the processed image frame may be displayed on the display unit 151; 0096].
Kim does not teach the instructions cause the refrigerator to control the camera based on detecting one of the first door or the second door is opened.
However, Takeshi teaches a method of processing image data in a refrigerator [0001] including where the instructions [where control SoC 41 issues a shooting instruction to camera 33; 0023] cause the refrigerator [refrigerator 100, Figure 1] to control the camera [image pickup device 30, Figure 1;0030] based on detecting one of the first door or the second door [refrigerator room doors 1a and 1b, Figure 1] is opened [when SoC41 detects the door 1 is opened; 0023] where one of ordinary skill in the art would have been capable of applying this known technique, ensuring the refrigerator door is open, to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art i.e., allowing the entire interior of the fridge to be captured [Takeshi; 0018].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the assembly of the combined teachings to have where the instructions cause the refrigerator to control the camera based on detecting one of the first door or the second door is opened in view of the teachings of Takeshi where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable i.e., allowing the entire interior of the fridge to be captured [Takeshi; 0018].
Regarding Claim 9, Kim, as modified, teaches the invention of claim 1 and does not teach the output device [output unit 150; 0062], wherein the one or more instructions [where one or more instructions constitute the learning model stored in memory 230; where AI device 100 may infer the result value for the input data using the learning model and generate the response or control command based on the result; 0079;0080], when executed by the at least one processor [processor 260 of AI server 200 and processor 180 of AI device 100, Figure 2 and Figure 4], further cause the refrigerator to:
rotate the optimal door bin image to be aligned with a horizontal line based on a conversion guideline, obtain an image of an area of interest by extracting the area of interest in the rotated optimal door bin image and cropping the area of interest, correct the image of the area of interest by applying a perspective transformation, and based on receiving a user input to output the optimal door bin image, control the output device to output the corrected image of the area of interest.
However, Takeshi teaches a method of processing image data in a refrigerator [0001] to rotate the optimal door bin image to be aligned [where microcomputer 52 performs an image rotation process to rotate the image of each storage section; 0041] with a horizontal line based on a conversion guideline [in the field of view from the front the viewing position of the user; 0041], obtain an image of an area of interest by extracting the area of interest in the rotated optimal door bin image [a food storage area; 0044] and cropping the area of interest [where the cropping process can remove at least a portion of the area that is not a food storage area from the input photographed image; 0044], correct the image of the area of interest by applying a perspective transformation [where microcomputer performs distortion correction in the depth direction to make it rectangular; 0042], and based on receiving a user input to output the optimal door bin image, control the output device to output the corrected image of the area of interest [where server 54 transmits fridge interior image data in response to a request to display a fridge interior image from a mobile device; 0024] where one of ordinary skill in the art would have been capable of applying this known technique, processing the door bin image, to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art i.e., providing an image of the interior of the refrigerator that is close to the state observed by the user while minimizing image switching operations on the display operation device. [Takeshi; 0060].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the assembly of the combined teachings to have wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: rotate the optimal door bin image to be aligned with a horizontal line based on a conversion guideline, obtain an image of an area of interest by extracting the area of interest in the rotated optimal door bin image and cropping the area of interest, correct the image of the area of interest by applying a perspective transformation, and based on receiving a user input to output the optimal door bin image, control the output device to output the corrected image of the area of interest in view of the teachings of Takeshi where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable i.e., providing an image of the interior of the refrigerator that is close to the state observed by the user while minimizing image switching operations on the display operation device. [Takeshi; 0060].
Regarding Claim 10, Kim, as modified, teaches the invention of claim 1 and further teaches wherein the one or more instructions [where one or more instructions constitute the learning model stored in memory 230; where AI device 100 may infer the result value for the input data using the learning model and generate the response or control command based on the result; 0079;0080], when executed by the at least one processor [processor 260 of AI server 200 and processor 180 of AI device 100, Figure 2 and Figure 4], further cause the refrigerator to:
obtain, based on the image of the area of interest, information on incoming groceries or outgoing groceries [where food from the image is detected using the object detection model;0244;0277-0279], and control the output device to provide information on the incoming groceries or the outgoing groceries [where the processor 180 displays food management information through the display 151 based on the acquired food storage state (S913);0278;0282].
Regarding Claim 11, Kim teaches a controlling method of a refrigerator [refrigerator 10, Figure 5A] comprising a body [cabinet 11, Figure 5A] comprising a storage chamber [refrigerating compartment 18, Figure 5A], a door [refrigerator door 12, Figure 5A ] comprising a door bin [where a storage space for accommodating foods may be in each of the inner door 13 and the outer door 14 of refrigerator door 12, Figure 5A; 0281] and rotatably coupled to the body to open or close the storage chamber [where door 12 is rotatably provided in front of the refrigerating compartment, Figure 5A;0113], and a camera on the body [camera 121 provided on a ceiling of cabinet 11, Figure 5B; 0120], the controlling method comprising:
based on detecting opening of the storage, controlling the camera to capture an image of at least a part of an inside of the body [where when opening of the storage rooms 510 and 530 is detected the processor 180 transmits a command for operation of camera 121 which may acquire an image of the first storage room 510 or second storage room 530;0169;0170];
identifying, based on the image, whether at least one of a first area of the inside of the body included in the image satisfies a predetermined condition [processor 180 extracts an image having the largest surface area of each of the storage rooms; 0182; 0183]; and based on identifying that the predetermined condition is satisfied, storing the image as an optimal door bin image [where the processed image frame may be stored in the memory 170;0096].
Kim does not explicitly teach controlling the camera to capture, based on detecting opening of the door, the image of the door.
However, Takeshi teaches a method of processing image data in a refrigerator [0001] where based on detecting opening of the door [when SoC41 detects the door is opened; 0023], the camera [image pickup device 30, Figure 1;0030] captures an image of an inside of the door [door storage section 11a and 11b, Figure 11;0038] where one of ordinary skill in the art would have been capable of applying this known technique, a wide-angle fisheye lens and ensuring the door is open, to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art i.e., allowing the entire interior of the fridge to be viewed [Takeshi; 0018].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of Kim to have controlling the camera to capture, based on detecting opening of the door, the image of the door in view of the teachings of Takeshi where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable i.e., allowing the entire interior of the fridge to be viewed [Takeshi; 0018].
Regarding Claim 12, Kim, as modified, teaches the method of claim 11 and further teaches wherein the camera is located on an upper end of the body, wherein the first area of the inside of the body is an area on a lower end and a side surface of the body covered by the door in a state in which the door is closed, and wherein the first line is on an upper end of the door [Refer to Kim in view of Takeshi as applied to the rejection of claim 2 above].
Regarding Claim 16, Kim teaches a non-transitory computer readable medium [a non-transitory computer readable medium, claim 15] having instructions stored therein [a program for causing a processor implemented in an artificial intelligence refrigerator to execute operations, claim 15] which when executed by a processor [processor 260 of AI server 200 and processor 180 of AI device 100, Figure 2 and Figure 4] of a refrigerator [refrigerator 10, Figure 5A] comprising a body [cabinet 11, Figure 5A] comprising a storage chamber [refrigerating compartment 18, Figure 5A], a door [refrigerator door 12, Figure 5A] comprising a door bin [where a storage space for accommodating foods may be in each of the inner door 13 and the outer door 14 of refrigerator door 12, Figure 5A; 0281] and rotatably coupled to the body to open or close the storage chamber [where door 12 is rotatably provided in front of the refrigerating compartment, Figure 5A;0113], and a camera located on the body [camera 121 provided on a ceiling of cabinet 11, Figure 5B; 0120], cause the processor to execute a method [where one or more instructions constitute the learning model stored in memory 230; where AI device 100 may infer the result value for the input data using the learning model and generate the response or control command based on the result; 0079;0080] comprising:
based on detecting opening of the storage, controlling the camera to capture an image of at least a part of an inside of the body [where when opening of the storage rooms 510 and 530 is detected the processor 180 transmits a command for operation of camera 121 which may acquire an image of the first storage room 510 or second storage room 530;0169;0170];
identifying, based on the image, whether at least one of a first area of the inside of the body included in the image satisfies a predetermined condition [where processor 180 extracts an image having the largest surface area of each of the storage rooms; 0182; 0183]; and
based on identifying that the predetermined condition is satisfied, storing the image as an optimal door bin image [where the processed image frame may be stored in the memory 170;0096].
Kim does not explicitly teach based on detecting opening of the door and controlling the camera to capture an image of an inside of the door.
However, Takeshi teaches a method of processing image data in a refrigerator [0001] including controlling the camera [image pickup device 30, Figure 1;0030] to capture an image of an inside of the door [door storage section 11a and 11b, Figure 11;0038], based on detecting opening of the door [when SoC41 detects the door is opened; 0023], where one of ordinary skill in the art would have been capable of applying this known technique, a wide-angle fisheye lens and ensuring the door is open, to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art i.e., allowing the entire interior of the fridge to be viewed [Takeshi; 0018].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the assembly of Kim to have based on detecting opening of the storage controlling the camera to capture an image of an inside of the door in view of the teachings of Takeshi where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable i.e., allowing the entire interior of the fridge to be viewed [Takeshi; 0018].
Regarding Claim 17, Kim, as modified, teaches the invention of claim 16 and further teaches where the camera is on an upper end of the body, wherein the first area of the inside of the body is an area on a lower end and a side surface of the body covered by the door in a state in which the door is closed, and wherein the first line is on an upper end of the door. [Refer to Kim in view of Takeshi as applied to claim 2 above].
PNG
media_image1.png
518
610
media_image1.png
Greyscale
Claims 3, 8, 13 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (US20230043011A1) in view of Takeshi et al. (WO2022038799A1) as applied to claim 1 above and in further view of Chung et al. (US20230375258A1).
Regarding Claim 3, Kim, as modified, teaches the invention of claim 1 and further teaches wherein the one or more instructions [where one or more instructions constitute the learning model stored in memory 230; where AI device 100 may infer the result value for the input data using the learning model and generate the response or control command based on the result; 0079;0080], when executed by the at least one processor [processor 260 of AI server 200 and processor 180 of AI device 100, Figure 2 and Figure 4], further cause the refrigerator to:
obtain a template image [where the processor 180 may store an image of the boundary area A of the first storage room 510 in advance; 0189] while the door is open and is halted during a specific time [where processor 180 acquires a photographing time point corresponding to the extracted image as the maximum opening time point; 0184], compare the template image with the image captured by the camera [where the processor may compare the image of boundary area A stored in the memory 170 to the obtained image; 0190], and identify, based on a result of the comparison of the template image with the image captured by the camera [where processor 180 recognizes the boundary area A; 0170], whether the first area of the inside of the body satisfy the predetermined condition [an area having the largest surface area; 0196].
Kim does not teach where the instructions cause the refrigerator to identify, based on a result of the comparison of the template image with the image captured by the camera, the first line included in the door satisfy the predetermined condition.
However, Chung teaches systems and methods for detecting a position of a door [0001] where the instructions [where the processor executes non-transitory programming instructions stored in memory; 0031] cause the refrigerator [refrigerator 100, Figure 1] to identify, based on a result of the comparison of the template image [where it may be determined in advance where the fiducial point should be; 0061] with the image captured by the camera [where the method includes an image capture sequence for analysis, evaluation, or comparison; 0056], the first line included in the door [where a fiducial point on the door is identified; 0060; where a fiducial point 210 may be identifiable by a counter line formed by an outer visible edge of the corresponding object on the door 112; 0038] satisfy the predetermined condition [where the fiducial point is compared to an intended location; 0061] where one of ordinary skill in the art would have been capable of applying this known technique, identifying a line included in the door for analysis, to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art i.e., isolating important object regions by edge matching [Chung, 0040].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the assembly of the combined teachings to have where the predefined temperature range includes based on a result of the comparison of the template image with the image captured by the camera, the first line included in the door satisfy the predetermined condition in view of the teachings of Chung where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable i.e., isolating important object regions by edge matching [Chung, 0040].
Regarding Claim 8, Kim, as modified, teaches the invention of claim 1 and further teaches where the camera is a wide-angle camera [the image capture device having a wide-angle fisheye lens (e.g., lens 34); 0034 of Takeshi; refer to Takeshi as applied to the rejection of Claim 1 above], and wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: identify whether the first area of the inside of the body included in the corrected image [where processor 180 deletes a lower area 1201-N of boundary area A and acquires an Nth remaining area 1203-N, also known as cropping; 0194-0196] satisfy the predetermined condition [where processor 180 extracts an image corresponding to an area having the largest surface area among the remaining areas 1203-1 to 1203-N, the cropped areas; 0194-0196] but Kim does not teach where wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to correct an image captured by the wide-angle camera.
However, Takeshi teaches a method of processing image data in a refrigerator [0001] to correct an image [where the control SoC 41 performs a fridge interior image correction process;0028] captured by the wide angle camera [imaging device 30 with a fisheye lens 34; 0015], where one of ordinary skill in the art would have been capable of applying this known technique, ensuring the refrigerator door is open, to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art i.e., allowing the entire interior of the fridge to be captured while providing an image close to the user’s line of sight [Takeshi; 0033].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the assembly of the combined teachings to have wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: correct an image captured by the wide angle camera in view of the teachings of Takeshi where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable i.e., allowing the entire interior of the fridge to be captured while providing an image close to the user’s line of sight [Takeshi; 0033].
Kim does not teach wherein the one or more instructions, when executed by the at least one processor, further cause the refrigerator to: identify whether the first line included in the door included in the corrected image satisfy the predetermined condition.
However, Chung teaches methods for detecting a position of the refrigerator door [0001] where one or more instructions [where the processor executes non-transitory programming instructions stored in memory; 0031], when executed by the at least one processor [controller 150; 0031], cause the refrigerator [refrigerator 100, Figure 1] to: identify whether the first line included in the door included in the image [where a fiducial point on the door is identified; 0060; where a fiducial point 210 may be identifiable by a counter line formed by an outer visible edge of the corresponding object on the door 112; 0038; ] satisfy the predetermined condition [where the fiducial point is compared to an intended location; 0061] where one of ordinary skill in the art would have been capable of applying this known technique, ensuring the refrigerator door is open, to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art i.e., isolating important object regions by edge matching [Chung, 0040].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the assembly of the combined teachings to have when executed by the at least one processor, further cause the refrigerator to: the first line included in the door included in the corrected image satisfy the predetermined condition in view of the teachings of Chung where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable i.e., isolating important object regions by edge matching [Chung, 0040].
Regarding Claim 13, Kim, as modified, teaches the method of claim 11 and further teaches wherein the identifying whether the image satisfies the predetermined condition comprises: obtaining a template image while the door is open and is halted during a specific time; comparing the template image with the image captured by the camera; and identifying, based on a result of the comparing, whether the first area of the inside of the body and the first line included in the door satisfy the predetermined condition. [Refer to Kim in view of Takeshi and Chung as applied to the rejection of claim 3 above.]
Regarding Claim 18, Kim, as modified, teaches the invention of claim 16 and further teaches wherein the identifying whether the image satisfies the predetermined condition comprises: obtaining a template image while the door is open and is halted during a specific time; comparing the template image with the image captured by the camera; and identifying, based on a result the comparing, whether the first area of the inside of the body and the first line included in the door satisfy the predetermined condition. [Refer to Kim in view of Takeshi and Chung as applied to the rejection of claim 3 above.]
Claims 4, 5, 14, 15, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (US20230043011A1) in view of Takeshi et al. (WO2022038799A1) and Chung et al. (US20230375258A1) as applied to claim 3 and 13 above and in further view of Guo et al. (CN114279367A)
Regarding Claim 4, Kim, as modified, teaches the invention of claim 3 and further teaches where the one or more instructions [where one or more instructions constitute the learning model stored in memory 230; where AI device 100 may infer the result value for the input data using the learning model and generate the response or control command based on the result; 0079;0080], when executed by the at least one processor [processor 260 of AI server 200 and processor 180 of AI device 100, Figure 2 and Figure 4], further cause the refrigerator to:
based on determining that the storage is in the state of being open and halted during the specific time [a time point corresponding to the maximum opening time; 0184], obtain, as the template image, an image at a time point in the state in which the door is open and halted [where the processor 180 may store an image of the boundary area A of the first storage room 510 in advance in a memory 170 to compare to the obtained image; 0189;0190].
Kim does not teach where the one or more instructions cause the refrigerator to detect a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera,
determine, based on the plurality of images, whether the door is in a moving state,
based on the change of the first area of the inside of the body, determine that the door is in a state of being open and halted, and
based on determining that the door is in the state of being open and halted during the specific time, obtain an image at a time point in the state in which the door is open and halted.
However Chung teaches including to detect a change of the first area of the inside of the body [where if the measured brightness value is compared to the predetermined brightness threshold and determined to be greater than the predetermined brightness threshold, it may be determined that the door is in the partially opened state; 0062] and the first line included in the door [where a fiducial point 210 may be identifiable by a counter line formed by an outer visible edge of the corresponding object on the door 112; 0038] included in a plurality of images captured by the camera [where at 640, the method 600 includes determining a shut state of the door based on the captured image(s) at 630; 0058], determine, based on the plurality of images [where the method 600 includes determining a shut state of the door based on the captured image(s) at 630; 0058], whether the door is in a moving state, based on the change of the first area of the inside of the body a [where it may be determined that an opening of the door is likely; 0058], determine that the door is in a state of being open [if the measured brightness value is compared to the predetermined brightness threshold and determined to be greater than the predetermined brightness threshold, it may be determined that the door is in the partially opened state; 0062] and halted [where an end condition may include failure to detect further changes in sequential images of the image capture sequence; 0057], and based on determining that the door is in the state of being open and halted during the specific time, obtain, an image at a time point in the state in which the door is open and halted [image capture sequence; 0057] where one of ordinary skill in the art would have been capable of applying this known technique, detecting a change in brightness value, to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art i.e., improving the accuracy of the image processing algorithm by determining the state of the door with additional variables [Chung, 0048].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the assembly of the combined teachings to have where the one or more instructions cause the refrigerator to detect a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera, determine, based on the plurality of images, whether the door is in a moving state, based on the change of the first area of the inside of the body, determine that the door is in a state of being open and halted, and based on determining that the door is in the state of being open and halted during the specific time, obtain an image at a time point in the state in which the door is open and halted in view of the teachings of Chung where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable i.e., improving the accuracy of the image processing algorithm by determining the state of the door with additional variables [Chung, 0048].
Kim further does not teach the instructions cause the refrigerator to: based on the first line included in the door while the door is open being smaller than or equal to a predetermined value, determine that the door is in a state of being open and halted.
However, Guo teaches a method for detecting the opening angle of a display case and a storage medium [n0001;n0002] including, based on the first line included in the door while the door is open [where the length of the bottom edge of the cabinet door is part of a triangle used to calculate the opening angle of the cabinet door, Figure 9; n0123-n0126] being equal to a predetermined value [where the target triangle is an equilateral triangle, such that the length of the bottom edge of the cabinet door is equal to the remaining side, Figure 9; n0126], to determine that the door is in a state of being open and halted [when the cabinet door reaches the target triangle; n0126; when the cabinet door opening angle has been opened to the maximum angle; n0131] where one of ordinary skill in the art would have been capable of applying this known technique to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art i.e., providing technical support for realizing other functions of the display cabinet based on the opening angle [Guo; n0031]
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the assembly of the combined teachings to have the instructions cause the refrigerator to: based on the first line included in the door while the door is open being smaller than or equal to a predetermined value, determine that the door is in a state of being open and halted in view of the teachings of Guo where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable i.e., providing technical support for realizing other functions of the display cabinet based on the opening angle [Guo; n0031]
Regarding Claim 5, Kim, as modified, teaches the invention of claim 3 and further does not teach where the one or more instructions [where one or more instructions constitute the learning model stored in memory 230; where AI device 100 may infer the result value for the input data using the learning model and generate the response or control command based on the result; 0079;0080], when executed by the at least one processor [processor 260 of AI server 200 and processor 180 of AI device 100, Figure 2 and Figure 4], further cause the refrigerator to: based on determining that a difference value between the first area included in the template image and the first area included in the image captured by the camera is a first predetermined value, or a length of the first line included in the image captured by the camera is greater than a second predetermined value, or a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value, determining that the predetermined condition is satisfied.
However, Guo teaches a method for detecting the opening angle of a display case [0003] where based on determining that a door angle [determine opening angle based on side lengths of targe triangle; n0124] obtained based on the first line [Based on the position information of the bottom edge of the cabinet and the bottom edge of the cabinet door, determine the lengths of each side of the target triangle; the bottom edge of the cabinet and the bottom edge of the cabinet door in the target image are each a side of the target triangle; n0123] included in the image captured by the camera is a third predetermined value [when the opening angle of the cabinet door meets the preset triggering requirements; n0129; where it is possible to detect whether the cabinet door opening angle has been opened to the maximum angle; n0131], determining that the predetermined condition is satisfied [where the second camera is triggered to take a picture of the inside of the display cabinet; n0129] where one of ordinary skill in the art would have been capable of applying this known technique to a known device that was ready for improvement and the results would have been predictable to one of ordinary skill in the art, i.e., providing technical support for realizing other functions of the display cabinet based on the opening angle [Guo; n0031].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the assembly of the combined teachings to where based on determining that a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value , determining that the predetermined condition is satisfied in view of the teachings of Guo where this known technique could have been applied to a known device that was ready for improvement and the results would have been predictable, i.e., providing technical support for realizing other functions of the display cabinet based on the opening angle [Guo; n0031].
Regarding Claim 14, Kim, as modified, teaches the invention of claim 13 and further teaches wherein the obtaining the template image comprises: detecting a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera; determining, based on the plurality of images, whether the door is in a moving state; based on the change of the first area of the inside of the body and the first line included in the door while the door is open being smaller than or equal to a predetermined value, determining that the door is in a state of being open and halted; and based on determining that the door is in the state of being completely and halted during the specific time, obtaining, as the template image, an image at a time point in the state the door was completely open and halted. [Refer to Kim in view of Takeshi, Chung and Guo as applied to the rejection of claim 4 above].
Regarding Claim 15, Kim, as modified, teaches the invention of claim 13 and further teaches where the identifying whether the predetermined condition is satisfied comprises: based on determining that a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value, identifying that the predetermined condition is satisfied [Refer to Kim in view of Takeshi, Chung and Guo as applied to the rejection of claim 5 above].
Regarding Claim 19, Kim, as modified, teaches the invention of claim 18 and further teaches where the obtaining the template image comprises: detecting a change of the first area of the inside of the body and the first line included in the door included in a plurality of images captured by the camera; determining, based on the plurality of images, whether the door is in a moving state; based on the change of the first area of the inside of the body and the first line included in the door while the door is open being smaller than or equal to a predetermined value, determining that the door is in a state of being open and halted; and based on determining that the door is in the state of being completely and halted during the specific time, obtaining, as the template image, an image at a time point in the state the door was completely open and halted. [Refer to Kim in view of Takeshi, Chung and Guo as applied to the rejection of claim 4 above].
Regarding Claim 20, Kim, as modified, teaches the invention of claim 18 and further teaches wherein the identifying whether the predetermined condition is satisfied comprises: based on determining a door angle obtained based on the first line included in the image captured by the camera is a third predetermined value, determining that the predetermined condition is satisfied. [Refer to Kim in view of Takeshi, Chung and Guo as applied to the rejection of claim 5 above].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEONA LAUREN BANKS whose telephone number is (571)270-0426. The examiner can normally be reached Mon-Fri 8:30- 6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jerry-Daryl Fletcher can be reached at 5712705054. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEONA LAUREN BANKS/Examiner, Art Unit 3763
/ELIZABETH J MARTIN/Primary Examiner, Art Unit 3763