Dev (Sourcery refactored) - #18
Conversation
290fbe5 to
bfcfdbfCompare| def greet(name): | ||
| print('Hello {}'.format(name)) | ||
| print(f'Hello {name}') |
There was a problem hiding this comment.
Function greet refactored with the following changes:
- Replace call to format with f-string (
use-fstring-for-formatting)
| """ | ||
| def job1(): | ||
| print("I'm running on threads %s" % threading.current_thread()) | ||
| print(f"I'm running on threads {threading.current_thread()}") |
There was a problem hiding this comment.
Function job1 refactored with the following changes:
- Replace interpolated string formatting with f-string (
replace-interpolation-with-fstring)
| print(f"I'm running on threads {threading.current_thread()}") | ||
| def job2(): | ||
| print("I'm running on threads %s" % threading.current_thread()) | ||
| print(f"I'm running on threads {threading.current_thread()}") |
There was a problem hiding this comment.
Function job2 refactored with the following changes:
- Replace interpolated string formatting with f-string (
replace-interpolation-with-fstring)
| print(f"I'm running on threads {threading.current_thread()}") | ||
| def job3(): | ||
| print("I'm running on threads %s" % threading.current_thread()) | ||
| print(f"I'm running on threads {threading.current_thread()}") |
There was a problem hiding this comment.
Function job3 refactored with the following changes:
- Replace interpolated string formatting with f-string (
replace-interpolation-with-fstring)
| CHROME_DRIVER_MAPPING_FILE = r"{}\mapping.json".format(CHROME_DRIVER_FOLDER) | ||
| CHROME_DRIVER_EXE = r"{}\chromedriver.exe".format(CHROME_DRIVER_FOLDER) | ||
| CHROME_DRIVER_ZIP = r"{}\chromedriver_win32.zip".format(CHROME_DRIVER_FOLDER) | ||
| CHROME_DRIVER_MAPPING_FILE = f"{CHROME_DRIVER_FOLDER}\mapping.json" | ||
| CHROME_DRIVER_EXE = f"{CHROME_DRIVER_FOLDER}\chromedriver.exe" | ||
| CHROME_DRIVER_ZIP = f"{CHROME_DRIVER_FOLDER}\chromedriver_win32.zip" |
There was a problem hiding this comment.
Lines 13-15 refactored with the following changes:
- Replace call to format with f-string [×3] (
use-fstring-for-formatting)
| ret = [] | ||
| for i in range(1, page): | ||
| ret.append('http://www.samair.ru/proxy/proxy-%(num)02d.htm' % {'num': i}) | ||
| return ret | ||
| return [ | ||
| 'http://www.samair.ru/proxy/proxy-%(num)02d.htm' % {'num': i} | ||
| for i in range(1, page) | ||
| ] |
There was a problem hiding this comment.
Function build_list_urls_3 refactored with the following changes:
- Convert for loop into list comprehension (
list-comprehension) - Inline variable that is immediately returned (
inline-immediately-returned-variable)
| ip = match[0] + "." + match[1] + match[2] | ||
| ip = f"{match[0]}.{match[1]}{match[2]}" |
There was a problem hiding this comment.
Function parse_page_3 refactored with the following changes:
- Use f-string instead of string concatenation [×3] (
use-fstring-for-concatenation)
| ret = [] | ||
| for i in range(1, page): | ||
| ret.append('http://www.pass-e.com/proxy/index.php?page=%(n)01d' % {'n': i}) | ||
| return ret | ||
| return [ | ||
| 'http://www.pass-e.com/proxy/index.php?page=%(n)01d' % {'n': i} | ||
| for i in range(1, page) | ||
| ] |
There was a problem hiding this comment.
Function build_list_urls_4 refactored with the following changes:
- Convert for loop into list comprehension (
list-comprehension) - Inline variable that is immediately returned (
inline-immediately-returned-variable)
| ret = [] | ||
| for i in range(1, page): | ||
| ret.append('http://www.ipfree.cn/index2.asp?page=%(num)01d' % {'num': i}) | ||
| return ret | ||
| return [ | ||
| 'http://www.ipfree.cn/index2.asp?page=%(num)01d' % {'num': i} | ||
| for i in range(1, page) | ||
| ] |
There was a problem hiding this comment.
Function build_list_urls_5 refactored with the following changes:
- Convert for loop into list comprehension (
list-comprehension) - Inline variable that is immediately returned (
inline-immediately-returned-variable)
| ret = [] | ||
| for i in range(1, page): | ||
| ret.append('http://www.cnproxy.com/proxy%(num)01d.html' % {'num': i}) | ||
| return ret | ||
| return [ | ||
| 'http://www.cnproxy.com/proxy%(num)01d.html' % {'num': i} | ||
| for i in range(1, page) | ||
| ] |
There was a problem hiding this comment.
Function build_list_urls_6 refactored with the following changes:
- Convert for loop into list comprehension (
list-comprehension) - Inline variable that is immediately returned (
inline-immediately-returned-variable)
| type = -1 # 该网站未提供代理服务器类型 | ||
| for match in matches: | ||
| ip = match[0] | ||
| port = match[1] | ||
| type = -1 # 该网站未提供代理服务器类型 |
There was a problem hiding this comment.
Function parse_page_6 refactored with the following changes:
- Hoist statements out of for/while loops (
hoist-statement-from-loop)
| ret = [] | ||
| for i in range(0, page): | ||
| ret.append('http://proxylist.sakura.ne.jp/index.htm?pages=%(n)01d' % {'n': i}) | ||
| return ret | ||
| return [ | ||
| 'http://proxylist.sakura.ne.jp/index.htm?pages=%(n)01d' % {'n': i} | ||
| for i in range(0, page) | ||
| ] |
There was a problem hiding this comment.
Function build_list_urls_9 refactored with the following changes:
- Convert for loop into list comprehension (
list-comprehension) - Inline variable that is immediately returned (
inline-immediately-returned-variable)
| if (type == 'Anonymous'): | ||
| type = 1 | ||
| else: | ||
| type = -1 | ||
| type = 1 if (type == 'Anonymous') else -1 |
There was a problem hiding this comment.
Function parse_page_9 refactored with the following changes:
- Replace if statement with if expression (
assign-if-exp)
| ret = [] | ||
| for i in range(1, page): | ||
| ret.append('http://www.publicproxyservers.com/page%(n)01d.html' % {'n': i}) | ||
| return ret | ||
| return [ | ||
| 'http://www.publicproxyservers.com/page%(n)01d.html' % {'n': i} | ||
| for i in range(1, page) | ||
| ] |
There was a problem hiding this comment.
Function build_list_urls_10 refactored with the following changes:
- Convert for loop into list comprehension (
list-comprehension) - Inline variable that is immediately returned (
inline-immediately-returned-variable)
| ret = [] | ||
| for i in range(1, page): | ||
| ret.append('http://www.my-proxy.com/list/proxy.php?list=%(n)01d' % {'n': i}) | ||
| ret.append('http://www.my-proxy.com/list/proxy.php?list=s1') | ||
| ret.append('http://www.my-proxy.com/list/proxy.php?list=s2') | ||
| ret.append('http://www.my-proxy.com/list/proxy.php?list=s3') | ||
| ret = [ | ||
| 'http://www.my-proxy.com/list/proxy.php?list=%(n)01d' % {'n': i} | ||
| for i in range(1, page) | ||
| ] | ||
| ret.extend( | ||
| ( | ||
| 'http://www.my-proxy.com/list/proxy.php?list=s1', | ||
| 'http://www.my-proxy.com/list/proxy.php?list=s2', | ||
| 'http://www.my-proxy.com/list/proxy.php?list=s3', | ||
| ) | ||
| ) |
There was a problem hiding this comment.
Function build_list_urls_11 refactored with the following changes:
- Merge consecutive list appends into a single extend [×2] (
merge-list-appends-into-extend) - Convert for loop into list comprehension (
list-comprehension)
| urls = [] | ||
| for page in range(beg, end): | ||
| urls.append('http://www.baidu.com?&page=%d' % page) | ||
| urls = ['http://www.baidu.com?&page=%d' % page for page in range(beg, end)] |
There was a problem hiding this comment.
Function run_spider refactored with the following changes:
- Convert for loop into list comprehension (
list-comprehension)
| res = [] | ||
| for i in range(len(s) - 1): | ||
| res.append((s[i], s[i + 1])) | ||
| res = [(s[i], s[i + 1]) for i in range(len(s) - 1)] |
There was a problem hiding this comment.
Function main refactored with the following changes:
- Convert for loop into list comprehension (
list-comprehension)
| proxies = { | ||
| 'http': '{proxy_type}://{ip}:{port}'.format(proxy_type=proxy_type, ip=ip, port=port), | ||
| 'https': '{proxy_type}://{ip}:{port}'.format(proxy_type=proxy_type, ip=ip, port=port), | ||
| return { | ||
| 'http': '{proxy_type}://{ip}:{port}'.format( | ||
| proxy_type=proxy_type, ip=ip, port=port | ||
| ), | ||
| 'https': '{proxy_type}://{ip}:{port}'.format( | ||
| proxy_type=proxy_type, ip=ip, port=port | ||
| ), | ||
| } | ||
| return proxies |
There was a problem hiding this comment.
Function get_proxy_dict refactored with the following changes:
- Inline variable that is immediately returned (
inline-immediately-returned-variable)
| urls = [] | ||
| for page in range(1, 1000): | ||
| urls.append('http://www.jb51.net/article/%s.htm' % page) | ||
| urls = [f'http://www.jb51.net/article/{page}.htm' for page in range(1, 1000)] |
There was a problem hiding this comment.
Function main refactored with the following changes:
- Convert for loop into list comprehension (
list-comprehension) - Replace interpolated string formatting with f-string (
replace-interpolation-with-fstring)
| class TestSyncSpider(SyncSpider): | ||
| def handle_html(self, url, html): | ||
| print(html) | ||
| pass |
There was a problem hiding this comment.
Function TestSyncSpider.handle_html refactored with the following changes:
- Remove redundant pass statement (
remove-redundant-pass)
| class TestAsyncSpider(AsyncSpider): | ||
| def handle_html(self, url, html): | ||
| print(html) | ||
| pass |
There was a problem hiding this comment.
Function TestAsyncSpider.handle_html refactored with the following changes:
- Remove redundant pass statement (
remove-redundant-pass)
| urls = [] | ||
| for page in range(1, 1000): | ||
| #urls.append('http://www.jb51.net/article/%s.htm' % page) | ||
| urls.append('http://www.imooc.com/data/check_f.php?page=%d'%page) | ||
| urls = [ | ||
| 'http://www.imooc.com/data/check_f.php?page=%d' % page | ||
| for page in range(1, 1000) | ||
| ] |
There was a problem hiding this comment.
Lines 24-27 refactored with the following changes:
- Convert for loop into list comprehension (
list-comprehension)
This removes the following comments ( why? ):
#urls.append('http://www.jb51.net/article/%s.htm' % page)
| func_name = 'get_' + field | ||
| func_name = f'get_{field}' | ||
| xpath_str = self.xpath_dict.get(field) | ||
| if hasattr(self, func_name): | ||
| return getattr(self, func_name)(xpath_str) | ||
| else: | ||
| self.logger.debug(field, self.url) | ||
| return self.parser.xpath(xpath_str)[0].strip() if xpath_str else '' | ||
| self.logger.debug(field, self.url) | ||
| return self.parser.xpath(xpath_str)[0].strip() if xpath_str else '' |
There was a problem hiding this comment.
Function XpathCrawler.get_field refactored with the following changes:
- Use f-string instead of string concatenation (
use-fstring-for-concatenation) - Remove unnecessary else after guard condition (
remove-unnecessary-else)
| xpath_result = {} | ||
| for field, xpath_string in self.xpath_dict.items(): | ||
| xpath_result[field] = makes(self.get_field(field)) # to utf8 | ||
| return xpath_result | ||
| return { | ||
| field: makes(self.get_field(field)) | ||
| for field, xpath_string in self.xpath_dict.items() | ||
| } |
There was a problem hiding this comment.
Function XpathCrawler.get_result refactored with the following changes:
- Convert for loop into dictionary comprehension (
dict-comprehension) - Inline variable that is immediately returned (
inline-immediately-returned-variable)
This removes the following comments ( why? ):
# to utf8
| category_urls = [] | ||
| for href in category_hrefs: | ||
| category_urls.append(urljoin(self.domain, href)) | ||
| category_urls = [urljoin(self.domain, href) for href in category_hrefs] |
There was a problem hiding this comment.
Function FantasyhairbuySite.generate_category_urls refactored with the following changes:
- Convert for loop into list comprehension (
list-comprehension)
| backup_name = os.path.basename(file_path) + '_' + datetime.datetime.now().strftime('%Y%m%d%H%M%S') | ||
| backup_name = ( | ||
| f'{os.path.basename(file_path)}_' | ||
| + datetime.datetime.now().strftime('%Y%m%d%H%M%S') | ||
| ) |
There was a problem hiding this comment.
Function backup_file refactored with the following changes:
- Use f-string instead of string concatenation (
use-fstring-for-concatenation)
| backup_list = glob.glob(os.path.join(path, pattern + '_*')) | ||
| backup_list = glob.glob(os.path.join(path, f'{pattern}_*')) |
There was a problem hiding this comment.
Function main refactored with the following changes:
- Use f-string instead of string concatenation (
use-fstring-for-concatenation)
| # if it's anything else, return it in its original form | ||
| return data | ||
| return data.encode('utf-8') if str(type(data)) == "<type 'unicode'>" else data |
There was a problem hiding this comment.
Function _byteify refactored with the following changes:
- Lift code into else after jump in control flow (
reintroduce-else) - Replace if statement with if expression (
assign-if-exp)
This removes the following comments ( why? ):
# if it's anything else, return it in its original form
| try: | ||
| res = query.find() | ||
| return res | ||
| return query.find() |
There was a problem hiding this comment.
Function LeanCloudApi.get_skip_obj_list refactored with the following changes:
- Inline variable that is immediately returned (
inline-immediately-returned-variable)
| img_info_url = img_url + '?imageInfo' | ||
| img_info_url = f'{img_url}?imageInfo' |
There was a problem hiding this comment.
Function LeanCloudApi.add_img_info refactored with the following changes:
- Use f-string instead of string concatenation (
use-fstring-for-concatenation)
Pull Request #17 refactored by Sourcery.
If you're happy with these changes, merge this Pull Request using the Squash and merge strategy.
NOTE: As code is pushed to the original Pull Request, Sourcery will
re-run and update (force-push) this Pull Request with new refactorings as
necessary. If Sourcery finds no refactorings at any point, this Pull Request
will be closed automatically.
See our documentation here.
Run Sourcery locally
Reduce the feedback loop during development by using the Sourcery editor plugin:
Review changes via command line
To manually merge these changes, make sure you're on the
devbranch, then run:Help us improve this pull request!