Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed on 6 November, 2023.
Response to Amendment
The amendment filed 26 March, 2026 has been entered.
The amendment of claims 1, 2, and 4 - 20 has been acknowledged.
Response to Arguments
Applicant’s arguments, see page 10, section “Rejections of Claims 1 – 3 and 15 – 18 Under 35 U.S.C. § 103”, filed 26 March, 2026 with respect to the rejection of claims 1 – 3 and 15 - 18 have been fully considered and are persuasive. The rejection of claims 1 – 3 and 15 - 18 have been withdrawn. However, upon further examination, a new rejection has been made under 35 U.S.C. § 103.
Claim Objections
Claim 5 is objected to because of the following informalities:
Claim 5 states “using selected at least one image pairs” in line 5, the examiner believes this should say “using the selected at least one image pairs” for clarity.
Claim 20 is objected to because of the following informalities:
Claim 20 states “using selected at least one image pairs” in line 7, the examiner believes this should say “using the selected at least one image pairs” for clarity.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 7, 8, and 12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 7 recites the limitation "the object to be synthesized" in line 5. There is insufficient antecedent basis for this limitation in the claim. The examiner believes this is intended to refer to “an object” in claim 6 in line 4 “performing a style transfer after synthesizing an object with at least one of the selected at least some image pairs”. The examiner believes “the object to be synthesized” is distinct from “an object” which has been synthesized with at least one of the selected at least some image pairs of claim 6. This is made more apparent by the claim limitation of claim 7 which states “synthesizing the object with a content image” in line 6 – 7 as it is unclear if “the object” refers to “the object to be synthesized” or “an object” which has been synthesized in claim 6.
Additionally, claim 7 states “in the at least one image pair” on line 8 and 10. It is unclear what this “at least one image pair” is intended to refer to. In line 5 – 6 the limitation reads “at least one of the selected at least some image pairs”, however this is not understood to explicitly be “the at least one image pair”.
Claim 8 states “selecting of the object to be synthesized with the at least one image pair” on line 5 - 6. There is insufficient antecedent basis for this limitation in the claim. Claim 7 recites the step which the examiner believes this limitation of claim 8 is further limiting, the step of “selecting, by the electronic device, the object to be synthesized with at least one of the selected at least one image pairs”. As written above regarding claim 7, It is unclear what this “at least one image pair” is intended to refer to as this is not understood to explicitly be the same as “the at least one of the selected at least some image pairs”.
Claim 12 states “the at least one illumination difference” on line 1. There is insufficient antecedent basis for this limitation in the claim. Claim 4 states “obtaining an illumination difference of at least one of the plurality of image pairs”, however there is no step of obtaining “at least one illumination difference”, instead it is “an illumination difference of at least one of the plurality of image pairs”.
Claim 12 states “the images” on line 8. It is not clear which images this intends to directly correspond to. Claim 12 describes “images included in the plurality of image pairs”, however claim 2 describes “images of the plurality of spots captured during a first time range”, “images of the plurality of spots captured during a second time range”; claim 4 describes “images having capturing spots corresponding to each other”; claim 7 describes “content images” and “style images”. It should be made clear which images described in claim 12 line 8 are referring to.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 3, and 15 - 18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al (U.S. Patent Publication No. 2023/0363610 A1, hereinafter “Zhang”) in view of Han et al (U.S. Patent Publication No. 2019/0294954 A1, hereinafter “Han”).
Regarding claim 1, Zhang teaches a method performed by an electronic device of training an object recognition model by using spatial information, the method comprising:
Obtaining, by the electronic device, the spatial information (¶ 0022: The method includes receiving image information captured by at least one visual spectrum-capable camera and location information captured by at least one depth measuring camera located on a mobile platform.)
Obtaining, by the electronic device, training data by using (¶ 0331: The image information is captured by at least one visual spectrum-capable camera that captures images in a visual spectrum (RGB) range and at least one depth measuring camera. The method includes generating at a time a training data, t0 set comprising 5000 to 10000 perception events.); and
Training, by the electronic device, a neural network model for object recognition by using the training data (¶ 0331: The method includes training a first set of perception classifier neural networks with the sensed information, object identification information, object shape information, and corresponding ground truth object identifications for each of the situational categories. The method includes saving parameters from training the perception classifier neural networks in tangible machine readable memory for use by the mobile platform in recognizing or responding to perceptions in the environment.).
Zhang does not explicitly teach the spatial information including illumination information corresponding to a plurality of spots in a space, wherein the illumination information includes information related to brightness of the plurality of spots; Obtaining, by the electronic device, at least one piece of the illumination information corresponding to at least one spot of the plurality of spots from the spatial information
However, Han does teach obtaining, by the electronic device, the spatial information including illumination information corresponding to a plurality of spots in a space, wherein the illumination information includes information related to brightness of the plurality of spots (¶ 0086: For example, in FIG. 4, the sensing unit 140 is shown having a proximity sensor 141 and an illumination sensor 142.; ¶ 0110: The sensing unit 140 is generally configured to sense one or more of internal information of the mobile terminal, surrounding environment information of the mobile terminal, user information, or the like.; ¶ 0120: If desired, an ultrasonic sensor may be implemented to recognize position information relating to a touch object using ultrasonic waves. The controller 180, for example, may calculate a position of a wave generation source based on information sensed by an illumination sensor and a plurality of ultrasonic sensors.);
Obtaining, by the electronic device, at least one piece of the illumination information corresponding to at least one spot of the plurality of spots from the spatial information (¶ 0175: As another example, when the AI service is the recognition service for autonomous movement and an illuminance of an image, a position of an object, a color of the object, a shape of the object, a direction of the object, or a relationship between the object and a background affects success or failure in providing the AI service, the feature to be extracted from the sensing data may include at least one of the illuminance of the image… (emphasis added));
Obtaining, by the electronic device, training data by using the obtained at least one piece of the illumination information and an image obtained by capturing the at least one spot (¶ 0181 If the principle of the VAE is applied to the present invention, the sensing data may be input to the encoder 720 of the AI device 10. In this case, the encoder 720 may compress the sensing data while maintaining the above described feature.; ¶ 0186: In this case, the decoder 740 may generate second sensing data from the feature data.); and
Training, by the electronic device, a neural network model for object recognition by using the training data (¶ 0024: Referring again to FIGS. 6 and 7, in step S630, the AI unit 230 may train a recognition model 750 for providing an AI service based on the second sensing data.; ¶ 0225: In detail, the AI unit 230 may provide the second sensing data as training data to train the recognition model 750. In this case, the recognition model 750 may learn a feature.).
Zhang and Han are considered to be analogous art as both pertain to mobile robot image analysis. Therefore, it would have been obvious to one of ordinary skill in the art to combine the occupancy map segmentation for autonomous guided platform with deep learning (as taught by Zhang) and the artificial intelligence server (as taught by Han) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Han utilizes historical information indicating the performing of a specific operation using machine learning and update existing learned information based on the analyzed information. This improves the information prediction accuracy of the predicted operation executed by the artificial intelligence unit (See ¶ 0082 - 0083).
This motivation for the combination of Zhang and Han is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim 2, the Zhang and Han combination teaches the method of claim 1.
Additionally, Zhang teaches wherein the obtaining of the spatial information includes:
obtaining, by the electronic device, images of the plurality of spots captured during a first time range (¶ 212: In order to track its location, the robot senses its own movement through understanding images captured by the depth sensing camera and RGB sensing camera, and one or more auxiliary sensor types (tactile, odometry, etc.)… The robot 2125 in FIG. 21 employs the cameras 202, 204 (of FIGS. 2A and 2B) of a multiple sensory input to capture image frames as well as distance (depth) information of the surrounding environment of room 2100.);
generating, by the electronic device, a map of the space by using the images captured during the first time range (¶ 0318: The method can include determining, by a processor, a 3D point cloud of points having 3D information including object location information (or object depth information or object distance information) from the depth measuring camera and the at least one visual spectrum-capable camera… The method can include using the 3D point cloud of points to prepare the occupancy map of the environment by locating the objects identified at locations in the 3D point cloud of points.); and
obtaining, by the electronic device, images of the plurality of spots captured during a second time range (¶ 212: In order to track its location, the robot senses its own movement through understanding images captured by the depth sensing camera and RGB sensing camera, and one or more auxiliary sensor types (tactile, odometry, etc.)… The robot 2125 in FIG. 21 employs the cameras 202, 204 (of FIGS. 2A and 2B) of a multiple sensory input to capture image frames as well as distance (depth) information of the surrounding environment of room 2100.; ¶ 0218: When the robot is activated in a previously mapped environment, the robot uses the technology described herein above in the Tracking sections to self-locate within the descriptive point cloud 2145.; Examiner's note: Tracking Sections appear to be ¶ 0212 - 0213).
Regarding claim 3, the Zhang and Han combination teaches the method of claim 2.
Additionally, Zhang teaches wherein a plurality of locations where capturing is performed during the first time range are recorded on the map (Figure 24B; ¶ 0216: Now with reference to Figure 22B, which illustrates an example of an occupancy grid 2250, the white portions indicate empty space – in other words space that has been determined from multiple sensory input to be unoccupied.) and
wherein the images captured during the second time range are captured during the second time range at the plurality of locations recorded on the map (¶ 0217: The descriptive point cloud 2145 and occupancy grid 2155 comprise a hybrid point grid that enables the robot 2125 to plan paths of travel through room 2100, using the occupancy grid 2155 and self-localize relative to features in the room 2100 using the descriptive point cloud 2145.; ¶ 212: In order to track its location, the robot senses its own movement through understanding images captured by the depth sensing camera and RGB sensing camera, and one or more auxiliary sensor types (tactile, odometry, etc.)… The robot 2125 in FIG. 21 employs the cameras 202, 204 (of FIGS. 2A and 2B) of a multiple sensory input to capture image frames as well as distance (depth) information of the surrounding environment of room 2100.; ¶ 0218: When the robot is activated in a previously mapped environment, the robot uses the technology described herein above in the Tracking sections to self-locate within the descriptive point cloud 2145.; Examiner's note: Tracking Sections appear to be ¶ 0212 - 0213).
Regarding claim 15, the Zhang and Han combination teaches the method of claim 1.
Additionally, Zhang teaches wherein at least some of parameters included in the neural network model are different from each other for each of the plurality of spots (¶ 0336: the training data set can further include images of different households.; ¶ 0337: The training data can further include images of at least one household environment containing a plurality of different furniture or barriers.; Examiner’s note: The process of retraining the model with new images would alter various parameters of the model with respect to the new images which are used for the retraining. As such the parameters would different with new photos being introduced to train the model.),
wherein the training of the neural network model includes additionally training the neural network model by using generated training data after the neural network model is firstly trained (¶ 0335: The method includes generating a third training data set at a time t1, later than time t0, including additional perception events reported after time t0. The method includes using the third training data set, performing the subdividing, training and saving steps to retrain the classifier neural networks, thereby enabling the classifiers to learn from subsequent activity.), and
wherein the additionally trained neural network model corresponds to a spot where an image used for generating of the training data used during the additional training is captured (¶ 0337: The training data set can further include images of at least one household environment containing a plurality of different furniture or barriers.; Examiners’ note: It is understood that additional training data for furniture and boundaries would be used to train the model to better recognize those areas with the furniture and boundaries where the images were collected.).
Regarding claim 16, claim 16 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Zhang’s further teaching on:
One or more non-transitory computer-readable storage media storing computer-executable instructions that, when executed by at least one processor of an electronic device individually or collectively, cause the electronic device to perform operations comprising (¶ 0323: The technology disclosed can include a non-transitory computer readable medium comprising stored instructions, which when executed by a processor, cause the processor to implement actions comprising the method presented above.)…
Regarding claim 17, claim 17 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Zhang’s further teaching on:
Memory, comprising one or more storage media, storing instructions and a program for training a neural network model (¶ 0564: A file storage subsystem 2936 can provide persistent storage for program and data files, and can include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations can be stored by file storage subsystem 2936 in the storage subsystem 2910, or in other machines accessible by the processor.); and
At least one processor communicatively coupled to the memory (¶ 0323: The technology disclosed can include a non-transitory computer readable medium comprising stored instructions, which when executed by a processor, cause the processor to implement actions comprising the method presented above.),
Wherein the instructions, when executed by the at least one processor individually or collectively, cause the computing device to (¶ 0323: The technology disclosed can include a non-transitory computer readable medium comprising stored instructions, which when executed by a processor, cause the processor to implement actions comprising the method presented above.)…
Regarding claim 18, claim 18 has been analyzed with regard to respective claim 2 and is rejected for the same reasons of obviousness as used above.
Allowable Subject Matter
Claims 4 – 13, 14, 19, and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ebrahimi Afrouzi et al (U.S. Patent No. 2020/0225673 A1) teaches a mobile robot that collects illumination and depth images to be used for object detection, and performing actions based on the detected object class.
Chae et al (U.S. Patent No. 2019/0370691 A1) teaches a cleaning mobile robot that determines cleaning routes using sensor data, the sensors comprising an image sensor and a depth sensor.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW JONES whose telephone number is (703)756-4573. The examiner can normally be reached Monday - Friday 8:00-5:00 EST, off Every Other Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW B. JONES/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights