프로그래밍 블로그

파이썬 크롤링 - 와이고수(Ygosu.com)

소스

import requests
from urllib import request
from bs4 import BeautifulSoup

BASE_URL = "https://www.ygosu.com/community/real_article"

# 헤더 설정
headers = [
{'User-Agent' : ''},
]

response = requests.get(BASE_URL, headers=headers[0])
soup = BeautifulSoup(response.content, 'html.parser')

html_list = soup.find('tbody').find_all('tr')

for img in html_list:
	category_list = img.find('td', class_='bdname').find('img')['title'] # 카테고리
	link_list = img.find('td', class_='tit').find('a', href=True)['href'] # 링크
	tile_list = img.find('td', class_='tit').find('a').text # 제목

	# print(category_list, tile_list+link_list)

출력 

print(category_list, tile_list + link_list)

엽게 남성연대 시위장면 https://www.ygosu.com/community/real_article/yeobgi/1819034/?page=0&frombest=Y
엽게 뭐 싸발아 눈깔아 ㅋㅋㅋㅋ https://www.ygosu.com/community/real_article/yeobgi/1819023/?page=0&frombest=Y
연예인 여름 https://www.ygosu.com/community/real_article/stars/149553/?page=0&frombest=Y
음식 안동가서 먹은것들 https://www.ygosu.com/community/real_article/food/75012/?page=0&frombest=Y
음식 구미 일품 칼국수 https://www.ygosu.com/community/real_article/food/75018/?page=0&frombest=Y
엽게 [ㅇㅎ] 포켓몬스터 - 알로라 디그다랑 스왑하기 https://www.ygosu.com/community/real_article/yeobgi/1819018/?page=0&frombest=Y
엽게 좆소회사 특징ㅋㅋhttps://www.ygosu.com/community/real_article/yeobgi/1819032/?page=0&frombest=Y
엽게 형;;  왼쪽 귀에서 피나요… https://www.ygosu.com/community/real_article/yeobgi/1819035/?page=0&frombest=Y
음식 연탄 고추장구이 https://www.ygosu.com/community/real_article/food/75011/?page=0&frombest=Y
엽게 군대 케이크 논란 반응 판 vs 여시  https://www.ygosu.com/community/real_article/yeobgi/1819029/?page=0&frombest=Y
농구 모란트 더블클러치 https://www.ygosu.com/community/real_article/basketball/28200/?page=0&frombest=Y
엽게 자기 와이프 난잡하게 먹어달라고해서 먹어준 썰.txt https://www.ygosu.com/community/real_article/yeobgi/1819001/?page=0&frombest=Y
엽게 식물갤 유저도 패드립하게 만든 상황.jpg https://www.ygosu.com/community/real_article/yeobgi/1818794/?page=0&frombest=Y
엽게 우주에서 가장 황당한 임신 https://www.ygosu.com/community/real_article/yeobgi/1818852/?page=0&frombest=Y
엽게 00년생 캐나다 누나 https://www.ygosu.com/community/real_article/yeobgi/1818913/?page=0&frombest=Y
엽게 그레이몬 https://www.ygosu.com/community/real_article/yeobgi/1819009/?page=0&frombest=Y
엽게 워그레이몬 https://www.ygosu.com/community/real_article/yeobgi/1819019/?page=0&frombest=Y
음식 막국수  https://www.ygosu.com/community/real_article/food/74931/?page=0&frombest=Y
음식 귀멸의 칼날 우동 만들어봤습니당 https://www.ygosu.com/community/real_article/food/74935/?page=0&frombest=Y
음식 개밥 https://www.ygosu.com/community/real_article/food/74952/?page=0&frombest=Y
음식 와붕이 저녁 https://www.ygosu.com/community/real_article/food/74956/?page=0&frombest=Y
음식 냠냠3 https://www.ygosu.com/community/real_article/food/74964/?page=0&frombest=Y
음식 쌀국수 https://www.ygosu.com/community/real_article/food/74977/?page=0&frombest=Y
엽게 에타 역지사지 레전드..jpg https://www.ygosu.com/community/real_article/yeobgi/1819013/?page=0&frombest=Y
엽게 취객 부축 레전드 https://www.ygosu.com/community/real_article/yeobgi/1819021/?page=0&frombest=Y