축구선수 몸값 분석 웹 크롤링 (2)

In [ ]:

### Transfermarkt 크롤링
첫 페이지와 두 번째 페이지 정보 크롤링 (총 50명)

In [1]:

import requests
from bs4 import BeautifulSoup
import pandas as pd
import time

headers = {'User-Agent' : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"}

player_list=[]
    
# url의 page 부분을 변수로 처리하여 반복문 실행
for i in range(1,3):
    url = f"https://www.transfermarkt.com/spieler-statistik/wertvollstespieler/marktwertetop?ajax=yw1&page={i}"

    r = requests.get(url, headers=headers)
    r.status_code

    # BeautifulSoup()으로 웹페이지 분석하기
    soup = BeautifulSoup(r.text, 'html.parser')
    # 선수들의 정보가 담긴 태그와 클래스 찾기
    player_info = soup.find_all('tr', class_ = ['odd','even'])

    # player_info에서 'td' 태그만 모두 찾기
    for info in player_info:
        player = info.find_all("td")
    # 해당 정보를 찾아서 각 리스트에 추가하기
        number = player[0].text 
        name = player[3].text 
        position = player[4].text 
        age = player[5].text 
        nation = player[6].img['alt'] 
        team = player[7].img['alt'] 
        value = player[8].text.strip()
        # 데이터에서 숫자만 가져오기 (인덱싱)
        value = value[1:-1] # ex. €160.00m 에서 €이 [0]이고 m이 [-1] 
        
        player_list.append([number, name, position, age, nation, team, value])
    
    time.sleep(1)  # 1페이지 당 sleep
        
    # pd.DataFrame()으로 저장하기
   
    
    df = pd.DataFrame(player_list, 
                   columns=['number', 'name', 'position', 'age', 'nation', 'team', 'value'])
   
# 저장된 파일 불러오기
# pd.read_csv('transfermarkt25.csv')

In [2]:

# 데이터 전처리(pre-processing)
df['value'] = df['value'].str.replace('€','')
df['value'] = df['value'].str.replace('m','').astype('float')
df['number'] = df['number'].astype('int64')
df['age'] = df['age'].astype('int64')
# df.head()
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 50 entries, 0 to 49
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   number    50 non-null     int64  
 1   name      50 non-null     object 
 2   position  50 non-null     object 
 3   age       50 non-null     int64  
 4   nation    50 non-null     object 
 5   team      50 non-null     object 
 6   value     50 non-null     float64
dtypes: float64(1), int64(2), object(4)
memory usage: 2.9+ KB

In [3]:

# 컬럼 생성 (한화 표시)
df['시장가치(억)'] = df['value']*13
df.head()

Out[3]:

	number	name	position	age	nation	team	value	시장가치(억)
0	1	Kylian Mbappé	Centre-Forward	23	France	Paris Saint-Germain	160.0	2080.0
1	2	Erling Haaland	Centre-Forward	21	Norway	Manchester City	150.0	1950.0
2	3	Vinicius Junior	Left Winger	21	Brazil	Real Madrid	100.0	1300.0
3	4	Phil Foden	Central Midfield	22	England	Manchester City	90.0	1170.0
4	5	Mohamed Salah	Right Winger	30	Egypt	Liverpool FC	90.0	1170.0

In [4]:

# 컬럼 삭제 df.drop(columns=['칼럼이름'])
df.drop(columns=['value'], inplace = True)  # inplace = True로 해야 저장됨!
df.head()

Out[4]:

	number	name	position	age	nation	team	시장가치(억)
0	1	Kylian Mbappé	Centre-Forward	23	France	Paris Saint-Germain	2080.0
1	2	Erling Haaland	Centre-Forward	21	Norway	Manchester City	1950.0
2	3	Vinicius Junior	Left Winger	21	Brazil	Real Madrid	1300.0
3	4	Phil Foden	Central Midfield	22	England	Manchester City	1170.0
4	5	Mohamed Salah	Right Winger	30	Egypt	Liverpool FC	1170.0

In [5]:

df.describe()

Out[5]:

	number	age	시장가치(억)
count	50.00000	50.000000	50.000000
mean	25.50000	24.400000	998.400000
std	14.57738	3.398679	237.743302
min	1.00000	17.000000	780.000000
25%	13.25000	22.000000	861.250000
50%	25.50000	24.000000	910.000000
75%	37.75000	27.000000	1040.000000
max	50.00000	31.000000	2080.000000

In [7]:

# df[칼럼 이름].mean()
# 나이평균 구하기
df['age'].mean()
# 몸값 합계 구하기 sum()
df['시장가치(억)'].sum()

Out[7]:

49920.0

In [8]:

# 선수들이 속한 가장 많은 나라는? 최빈값 mode()
df['nation'].mode()

Out[8]:

0    England
dtype: object

In [9]:

# 국적이 England인 선수들은?
df[df['nation']=='England']

Out[9]:

	number	name	position	age	nation	team	시장가치(억)
3	4	Phil Foden	Central Midfield	22	England	Manchester City	1170.0
5	6	Harry Kane	Centre-Forward	28	England	Tottenham Hotspur	1170.0
10	11	Jude Bellingham	Central Midfield	19	England	Borussia Dortmund	1040.0
11	12	Declan Rice	Defensive Midfield	23	England	West Ham United	1040.0
14	15	Trent Alexander-Arnold	Right-Back	23	England	Liverpool FC	1040.0
18	19	Jadon Sancho	Left Winger	22	England	Manchester United	975.0
19	20	Mason Mount	Attacking Midfield	23	England	Chelsea FC	975.0
32	33	Jack Grealish	Left Winger	26	England	Manchester City	910.0
35	36	Raheem Sterling	Left Winger	27	England	Manchester City	910.0
39	40	Bukayo Saka	Right Midfield	20	England	Arsenal FC	845.0
49	50	Reece James	Right-Back	22	England	Chelsea FC	780.0

In [10]:

# 데이터를 그룹으로 묶어 분석
df.groupby('nation')

Out[10]:

<pandas.core.groupby.generic.DataFrameGroupBy object at 0x0000024EF95F27C8>

In [11]:

g = df.groupby('nation')
g.size()

Out[11]:

nation
Argentina        1
Belgium          2
Brazil           3
Canada           1
Colombia         1
Egypt            1
England         11
France           2
Germany          6
Italy            2
Korea, South     1
Morocco          1
Netherlands      1
Nigeria          1
Norway           1
Portugal         6
Scotland         1
Senegal          1
Serbia           2
Slovakia         1
Spain            3
Uruguay          1
dtype: int64

In [13]:

g.count()

Out[13]:

	number	name	position	age	team	시장가치(억)
nation
Argentina	1	1	1	1	1	1
Belgium	2	2	2	2	2	2
Brazil	3	3	3	3	3	3
Canada	1	1	1	1	1	1
Colombia	1	1	1	1	1	1
Egypt	1	1	1	1	1	1
England	11	11	11	11	11	11
France	2	2	2	2	2	2
Germany	6	6	6	6	6	6
Italy	2	2	2	2	2	2
Korea, South	1	1	1	1	1	1
Morocco	1	1	1	1	1	1
Netherlands	1	1	1	1	1	1
Nigeria	1	1	1	1	1	1
Norway	1	1	1	1	1	1
Portugal	6	6	6	6	6	6
Scotland	1	1	1	1	1	1
Senegal	1	1	1	1	1	1
Serbia	2	2	2	2	2	2
Slovakia	1	1	1	1	1	1
Spain	3	3	3	3	3	3
Uruguay	1	1	1	1	1	1

In [15]:

# 수치형 데이터 총합 알아보기
g['시장가치(억)'].sum()

Out[15]:

nation
Argentina         975.0
Belgium          2015.0
Brazil           3185.0
Canada            910.0
Colombia          845.0
Egypt            1170.0
England         10855.0
France           3120.0
Germany          5395.0
Italy            1755.0
Korea, South      975.0
Morocco           845.0
Netherlands       910.0
Nigeria           845.0
Norway           1950.0
Portugal         5785.0
Scotland          845.0
Senegal           910.0
Serbia           2015.0
Slovakia          845.0
Spain            2860.0
Uruguay           910.0
Name: 시장가치(억), dtype: float64

In [19]:

# 퀴즈) 선수들의 몸값의 합이 큰 클럽별로 정렬해서 보여주기
result = df.groupby('team')
result['시장가치(억)'].sum().sort_values(ascending=False)

Out[19]:

team
Manchester City        9945.0
Bayern Munich          5395.0
Paris Saint-Germain    4810.0
Liverpool FC           3900.0
Inter Milan            3640.0
Juventus FC            2860.0
Chelsea FC             2665.0
Real Madrid            2210.0
Tottenham Hotspur      2145.0
Manchester United      2080.0
FC Barcelona           1820.0
RB Leipzig             1040.0
West Ham United        1040.0
Borussia Dortmund      1040.0
Bayer 04 Leverkusen     910.0
SS Lazio                910.0
Atlético de Madrid      910.0
AC Milan                910.0
Arsenal FC              845.0
SSC Napoli              845.0
Name: 시장가치(억), dtype: float64

In [ ]:

 

프로젝트에서 활용할 수 있는 부분)

• pandas와 groupby를 사용하면 사용자의 요구에 맞게 필요한 정보만 추출하여 제공할 수 있다.
• 그래프 등 시각적인 자료를 추가하여 선수들의 몸값 추이와 연령대 변화도 알 수 있다